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EXECUTIVE  SUMMARY 

This  study  was  performed  to  investigate  the  impact  of  hardware  and 
software  faults  on  system  reliability  and  to  develop  a  prediction  method¬ 
ology  that  is  applicable  early  in  the  development  life  cycle  when  it  is 
still  possible  to  influence  the  development  process  and  the  reliability  of 
the  resultant  system.  Since  hardware  reliability  prediction  is  relatively 
well  understood  and  practiced,  the  study  focused  on  software. 

Existing  software  reliability  prediction  models  rely  heavily  on  fault 
removal  rates  experienced  when  computer  software  is  exercised,  either  dur¬ 
ing  test  or  operational  usage.  Although  these  models  provide  meaningful 
results,  they  are  not  directly  applicable  during  early  development,  but 
rather  after  the  development  process  is  nearly  complete.  The  methodology 
presented  here  is  based  on  parameters  that  are  known  early  in  the  develop¬ 
ment  process.  As  the  product  evolves,  the  methodology  may  be  applied  iter¬ 
atively  using  progressively  more  accurate  input  parameters. 

This  approach  is  distinct  from  others  in  that  it  focuses  on  the 
process  used  to  develop  software,  as  well  as  the  unique  characteristics  of 
the  product  itself: 

1  Software  is  the  realization  of  an  algorithm  that  satisfies  a  given 
set  of  performance  and  functional  requirements.  Even  before  the 
software  exists,  its  requirements  are  known.  The  methodology  uses 
this  information  in  the  form  of  inherent  characteristics  of  the 
software , 

2_  Software  is  the  result  of  a  logical  process  in  the  collective  minds 
of  its  developers.  Unlike  hardware,  software  has  no  physical  parts 
to  fail.  Software  faults,  therefore,  are  the  result  of  errors  made 
during  the  development  process.  The  methodology  incorporates  error 
avoidance  and  detection  information  in  the  form  of  characteristics 
of  the  development  process. 

_3  Software  cannot  fail  unless  it  is  used.  Likewise,  software  com¬ 
ponents  cannot  fail  except  when  they  are  being  executed.  Logical 
paths  are  nonexistent  with  respect  to  reliability  when  they  are  not 
being  executed.  Conversely,  heavily  traveled  portions  of  the  soft¬ 
ware  tend  to  dominate  its  performance.  The  methodology  uses  a 
Markov  technique  to  determine  expected  software  performance  based 
on  mission-derived  path  probabilities. 

A  related  Rome  Air  Development  Center  (RADC)  study  [103]  is  currently 
being  performed  by  Software  Applications  International  Corporation  (SAIC). 
Whereas  the  objectives  of  the  Martin  Marietta  study  placed  emphasis  on 
reliability  prediction  at  the  beginning  of  the  development  life  cycle,  the 
SAIC  study  is  oriented  toward  the  end  of  the  development  phase  and  the 
beginning  of  the  system  integration  phase.  While  this  study  concentrates 
on  the  PROCESS  that  will  be  used  to  develop  the  product,  the  SAIC  study 
concentrates  on  the  PRODUCT  that  results  from  the  process.  Although  close 
liaison  between  studies  was  maintained  to  ensure  that  the  studies  didn't 
diverge  or  overlap,  the  methodologies  and  conclusions  drawn  were  inde¬ 
pendently  derived.  Interestingly,  the  methodologies  are  complementary. 
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The  development  life  cycle  of  any  system  involves  the  translation  of 
ideas  or  requirements  into  a  physical  product  by  means  of  some  engineering 
and/or  manufacturing  process.  In  the  beginning,  the  product  does  not 
exist,  but  its  functional  requirements  and  planned  development  process  arc 
knovm.  Reliability  prediction,  therefore,  must  be  accomplished  using  tnese 
known  parameters.  At  the  end,  the  product,  good  or  bad,  exists.  At  this 
time,  the  emphasis  can,  and  should,  focus  on  the  measurable  attributes  of 
the  product  itself.  Both  Martin  Marietta  and  SAIC  address  error  density 
within  a  software  product  as  a  prime  determinant  of  its  reliability.  The 
Martin  Marietta  study  focuses  on  predicting  error  density  based  on  inherent 
characteristics  of  the  product  and  the  error  avoidance  and  detection 
techniques  that  influence  it.  The  SAIC  study  has  defined  characteristic 
measures  of  the  developed  software  that  can  be  used  to  measure  its  error 
density. 

Other  models  exist  for  predicting  software  reliability  based  on  fault 
removal  rates  experienced  during  integration  testing  and  operational  usage. 
The  method  presented  herein  does  not  conflict  or  compete  with  them  either. 
Again,  the  differences  of  approach  are  due  to  the  development  phase  during 
which  the  prediction  technique  is  to  be  applied.  There  are  no  fault 
removal  rates  to  record  until  the  system  is  built  and  operated. 

The  overall  methodology  requires  the  prediction  of: 

Individual  component  (software  module)  success  probabilities 

^  Expected  path  probabilities  for  specific  operational  missions. 

A  method  is  described  to  predict  component  success  probabilities  based  on 
the  inherent  characteristics  of  the  component  and  the  process  that  will  be 
used  to  develop  it.  Likewise,  a  means  of  using  the  mission  scenario  to 
determine  path  probabilities  is  presented.  Finally,  the  overall  software 
system  reliability  is  predicted  by  combining  the  component  success  and  patli 
probabilities  mathematically. 

J_  At  the  beginning  of  the  development  life  cycle,  prediction  is  made 
based  on  the  development  process  as  described  herein. 

^  As  components  are  developed,  attributes  of  the  product  become 

measurable,  and  the  techniques  being  defined  by  SAIC  can  be  readily 
used  to  predict  component  probabilities  of  success.  In  addition, 
some  ut  Che  existing  reliability  models  can  also  be  applied  to 
measure  the  reliability  of  the  components.  The  same  methodology  is 
used,  except  that  one  of  Che  primary  parameters  becomes  measurable 
and  provides  more  accurate  inputs  to  the  prediction. 

_3  As  the  system  is  integrated  and  tested  in  a  real  or  simulated 

operational  environment,  the  path  probabilities  become  measurable. 
Again,  the  methodology  is  still  applicable  but  h.i  s  more  precise 
parameters . 
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While  the  study  has  addressed  many  aspects  of  the  prediction  problem, 
It  also  revealed  several  areas  where  more  detailed  research  and  data 
collection  is  needed: 

1  Demonstration  of  the  methodology  was  accomplished  using  error 
avoidance  and  detection  probabilities  statistically  derived  from 
the  two  surveys  conducted  during  the  study.  No  adequate  and 
complete  historical  data  base  could  be  found  to  quantitatively 
describe  the  effectiveness  of  the  error  detection  techniques  so 
highly  regarded  (qualitatively)  in  the  industry.  Although  nearly 
everyone  agrees  that  a  code  walkthrough  is  a  good  error  detection 
technique,  available  data  is  insufficient  to  determine  how  good. 
There  is  a  tremendous  void  of  good  historical  process  data. 

2  Extremely  high  execution  rates  of  computer  programs  allow  programs 
to  complete  thousands  of  executions  in  a  short  period  of  time.  If 
every  execution  is  cotisidered  to  be  an  independent  trial,  the  soft¬ 
ware  reliability  for  that  short  period  (thousands  of  executions) 
would  be  extremely  low  unless  the  single  execution  reliability  was 
infinitesimally  close  to  unity.  Fortunately,  a  logic  path  that 
works  correctly  one  time  will  always  work  correctly  if  it  is 
executed  with  identical  input  and  state  variables.  Of  course,  even 
a  simple  computer  program  is  exposed  to  a  nearly  infinite  number  of 
input  and  state  variable  combinations.  More  detailed  analysis  of 
input  domain  partitioning  is  needed  before  we  can  fully  understand 
and  evaluate  the  effects  of  continuous  cycling  of  software. 

There  is  a  school  of  thought  that  says  Chat  software  reliability  is 
dominated  by  rarely  used  paths.  The  rationale  is  that,  in  an 
operational  environment,  input  combinations  are  encountered  which 
cause  path  traversals  Chat  contain  errors  which  have  never  before 
been  exercised  during  test.  Although  the  methodology  does  not 
currently  utilize  path  probabilities  within  individual  software 
components,  it  can  be  easily  modified  as  our  knowledge  of  rarely 
used  paths  increases. 
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1.0  INTRODUCTION 


l.l  Overview  and  Objectives 

System  reliability  characterization  and  prediction  techniques,  which 
incorporate  the  combined  effects  of  both  hardware  and  software  components 
and  which  reflect  frequency  of  total  system  failures  during  mission  or 
operating  time,  are  needed  by  U.S.  Air  Force  systems  planners  and  procure¬ 
ment  offices.  Most  of  the  techniques  currently  available  for  predicting 
the  software  contributions  to  overall  system  reliability  rely  on  knowledge 
of  historical  failure-versus-t ime  data  to  project  future  performance. 
Although  such  methods  can  be  effectively  applied  during  the  testing  or 
operational  phase  of  a  system  life  cycle,  they  are  of  limited  value  during 
early  development  when  alternate  design  and  testing  strategies  can  be  eval¬ 
uated.  The  methodology  developed  during  this  study  presents  a  mechanism 
for  predicting  software  reliability  based  on  the  inherent  characteristics 
of  the  software  and  the  process  used  to  develop  it.  Coiiibined  with  e-xisting 
prediction  techniques  for  hardware  reliability,  this  technique  will  enable 
systems  planners  and  developers  to  assess  overall  system  reliability  at  a 
time  when  it  can  still  be  influenced. 

A  system  can  be  defined  as  a  collection  of  all  equipment,  facilities, 
procedures,  and  personnel  that  together  accomplish  a  specific  mission  or 
task.  Computers  are  an  integral  part  of  most  modern  systems  and  must  be 
included  in  any  system  analysis  process.  While  computer  reliability  can  be 
determined  by  classical  hardware  reliability  prediction  techniques,  the 
software  embedded  within  it  presents  a  new  challenge  to  analysts  who  must 
account  for  its  effects  on  the  system. 

Virtually  every  system  in  both  current  and  planned  military  inventory 
has  made  or  will  make  extensive  use  of  computer  technology.  In  nearly  all 
instances,  the  computer  and  its  embedded  software  are  not  only  part  of  the 
system,  they  are  essential  to  the  performance  of  required  operational  mis¬ 
sions.  Military  systems  can  be  distinguished  from  many  other  computerized 
systems  by  considering  some  of  their  special  characteristics: 

Operational  missions  involve  wartime  activities  that  cannot  be 
thoroughly  exercised  in  their  true  operational  environment. 

^  Overall  military  tactical  and  strategic  plans  and  decisions  require 
accurate  estimates  of  the  probability  of  success  of  individual 
systems . 

^  Each  system  must  be  available  when  needed  and  must  operate  correct¬ 
ly  from  the  time  it  is  activated  until  it  completes  its  mission. 

In  light  of  the  special  considerations  listed  above,  it  is  essential 
that  the  prediction  methodology  encompass  the  entire  system.  That  is,  all 
system  components  must  be  accounted  tor  within  the  technique.  Overall 
systems  reliability  is,  therefore,  defined  in  terms  of  its  missions  and  the 


specific  functions  it  is  required  to  perform.  The  following  definition  is 
assumed  throughout  this  report: 

SYSTEM  RELIABILITY  -  The  probability  that  tlie  required  system  will 
perform  its  intended  functions  for  prescribed  mission(s)  and  time 
period(s)  in  the  specified  operating  environment. 

Therefore,  the  objectives  of  the  study  were  to  develop 
characterization  techniques  for  predicting  total  system  reliability  and  to 
develop  a  practical  methodology  for  use  by  system  reliability  engineers 
early  in  the  development  cycle  to  evaluate  various  design  approaches  and 
alternatives  against  system  quantitative  reliability  requirements. 
Recognizing  the  inherent  differences  between  hardware  and  software,  the 
methodology  was  envisioned  as  consisting  of  separable,  but  analogous, 
prediction  techniques  for  each  and  a  means  for  combining  the  results  inti)  a 
single  prediction. 

Figure  1  illustrates  these  objectives.  Specific  attributes  affecting 
system  reliability  were  to  be  characterized  and  used  to  evaluate  hardware 
and/or  software  reliability.  The  results  would  then  be  combined  into  a 
single  reliability  prediction  for  the  entire  system. 

As  illustrated  in  Figure  1,  the  combined  system  reliability  model 
assumes  that  separate  methods  may  be  applied  to  hardware  and  software. 
Specifically,  the  method  assumes  that  physical  components  (hardware)  and 
logical  components  (software)  may  be  evaluated  independently  and  then 
combined  as  equally  essential  system  components.  That  is,  the  sysLitm 
reliability  detinition  given  earlier  can  be  expressed  as  a  function  of 
system  hardware  reliability  and  system  software  reliability: 

SYSTEM  HARDWARE  RELIABILITY  -  The  probability  that  the  required  hard¬ 
ware  will  perform  its  intended  functions  for  prescribed  missiim(s)  and 
lime  period(s)  in  the  specified  operating  environment,  without  causing 
system  outage  or  failure. 

SYSTEM  SOITWARE  RELIABILITY  -  The  probability  that  required  soltware 
will  perform  its  intended  functions  tor  prescribed  mission(s)  and  lime 
period(s)  in  the  specified  operating  environment,  witlioul  causing 
system  outage  or  failure. 

Since  hardware  reliability  techniques  have  been  well  developed  and 
applied,  the  study  primarily  concentrated  on  the  creation  of  a  software 
reliability  prediction  methodology.  The  approach  taken  was  to  concent ral e 
on  the  inherent  characteristics  of  the  software  and  its  development  pro¬ 
cess.  Hardware  aspects  of  the  problem  were  investigated  only  for  situa¬ 
tions  where  software  is  interfaced. 

1 . 2  Approach 

The  approach  taken  during  this  effort  involved  l '.le  per  f  orm.inci'  of  fiv.- 
interrelated  subtasks;  a  literature  search,  the  characterization  ol 
reliability  factors,  a  two-pass  survey,  the  development  of  a  syi.ti-m 


prediction  methodology,  and  the  validation  of  this  method  via  comparison 
with  data  collected  from  existing  systems. 

The  literature  search  was  used  to  quickly  assess  the  state  of  the  art 
with  respect  to  techniques  presently  available  and  to  identify  specific 
software  factors  and  characteristics  that  have  a  direct  or  indirect  effect 
on  system  reliability.  During  the  literature  search,  use  was  made  of 
several  in-house  and  external  data  bases  by  a  team  of  Martin  Marietta 
engineers,  representing  the  Reliability,  Systems,  Software,  and  Product 
Assurance  Departments. 

During  and  following  the  literature  search,  the  many  factors  that  were 
identified  had  to  be  evaluated  for  their  applicability  to  the  planned 
prediction  methodology.  Factors  were  categorized  from  several  different 
perspectives  to  determine  those  which  had  a  significant  impact  on  reli¬ 
ability  and  those  that  had  little  or  no  impact. 

The  surveys  were  used  to  solicit  and  summarize  the  subjective  opinions 
of  expert  practitioners  and  to  use  the  results  in  the  development  of  the 
methodology.  A  pilot  survey  was  created  and  distributed  based  on  the 
results  of  the  literature  search,  and  its  results  were  used  to  identify  and 
isolate  the  most  significant  software  factors  and  development  techniques. 

A  second  survey  based  on  the  results  of  the  pilot  and  the  requirements  of 
the  envisioned  methodology  was  distributed  to  a  much  wider  audience  and  was 
used  to  determine  the  quantitative  values  needed  by  the  methodology. 

Methodology  development  was  to  be  accomplished  by  the  identification, 
modification  and  application  of  existing  techniques  where  possible  and  by 
the  creation  of  new  techniques  where  necessary.  The  intent  was  to  produce 
a  software  reliability  prediction  technique  that  was  compatible  with  hard¬ 
ware  prediction  methods  and  that  could  be  readily  combined  with  hardware 
reliability  estimates. 

Concurrent  with  development  of  the  methodology,  real  data  from  oper¬ 
ational  military  systems  was  collected.  The  data  was  to  be  used  to  val¬ 
idate  the  methodology  by  comparing  the  actual  reliability  measured  for 
those  systems  to  the  reliability  predicted  by  the  methodology. 

1.3  Re  suit  s 

More  than  500  abstracts  and  100  documents  were  reviewed  during  the 
literature  search.  An  on-line  data  base  was  constructed  and  used  as  a 
reference  throughout  the  study.  Most  of  the  existing  software  models  were 
reviewed,  and  many  software  attributes  were  identified  and  considered  for 
inclusion  in  the  method. 

In  the  process  of  reviewing  the  factors  identified  during  the  litera¬ 
ture  search,  it  was  found  that  software  attributes  could  be  viewed  from 
various  perspectives.  Many  of  the  factors  were,  in  reality,  attributes  of 
an  existing  software  product  (lines  of  code,  number  of  branches,  etc.). 
Although  such  factors  can  be  used  for  prediction  purposes,  quantitative 
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relationships  between  the  given  attributes  (e.g.,  number  of  branches)  and 
reliability  were  found  to  be  non-existent  or  extremely  difficult  to  quan¬ 
tify  using  available  data.  Other  factors  were  found  to  be  characteristics 
of  the  software,  related  to  quality  considerations  rather  than  reliability. 
Such  factors  as  readability  and  flexibility  are  desirable  quality  character¬ 
istics,  but  do  not  directly  affect  the  ability  of  the  software  to  perform 
its  intended  functions. 

A  pilot  survey  was  conducted  to  provide  a  ranking  of  the  relative 
impact  of  each  factor  on  reliability.  All  of  the  factors  which  were 
identified  during  the  literature  search,  including  those  that  were  felt  to 
be  unrelated  to  reliability  prediction,  were  included.  The  survey  was 
distributed  to  thirty  hand-picked  software  experts  who  were  asked  to  rank 
them.  Results  were  used  to  eliminate  many  factors  from  further  considera¬ 
tion.  They  were  also  used  as  the  basis  for  a  full-scale  survey,  which  was 
distributed  to  more  than  300  software  practitioners.  In  the  full-scale 
survey,  respondents  were  asked  to  quantitatively  evaluate  each  of  the 
factors  listed  with  respect  to  its  influence  on  software  reliability. 
Approximately  100  persons  responded;  the  results  were  compiled  and  incor¬ 
porated  into  the  methodology. 

The  literature  search  also  revealed  that,  although  many  models  exist, 
nearly  all  are  concerned  with  the  software  reliability  measurement  and 
prediction  by  extrapolation.  Furthermore,  most  published  works,  concerning 
software  development  factors  and  techniques,  address  the  qualitative  rather 
than  quantitative  aspects  of  software.  Development  of  the  methodology, 
therefore,  required  that  a  new  technique  be  devised.  Wherever  possible, 
the  methodology  uses  approaches  and  techniques  developed  by  other 
researchers.  The  basic  features  of  the  methodology  are: 

Applicable  during  early  design/development 

^  Applicable  throughout  the  development  cycle 

3  Yields  quantitative  reliability  predictions 

^  Utility  as  a  design  and  process  evaluation  tool 

^  Uses  the  operational  mission  scenario  as  a  basis  for  prediction 

^  Compatible  with  MIL-HDBK-21 7D  techniques  and  reliability  defini- 
t ions . 

A  complete  validation  of  the  methodology  could  not  be  precisely 
accomplished  with  existing  reliability  data.  Although  extensive  amounts  of 
real  data  was  collected  during  the  effort,  the  completeness  of  the  data  was 
found  to  be  insufficient  for  making  comparisons  with  the  prediction  results. 
The  methodology  relates  inherent  functional  characteristics  and  development 
strategies  to  predicted  performance.  Historical  data  bases  tend  to  include 
only  failure  histories,  not  detailed  developmental  characteristics.  Current 
and  recently  completed  projects  do  not  yet  have  an  established  data  base  of 
failure  information.  The  methodology  was,  therefore,  validated  by  qualita¬ 
tive  comparisons  to  an  actual  system  and  through  use  of  several  detailed 
examples . 
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2.0  LITlSRATURt  SEARCH 


2.1  Overview  and  Objectives 

The  primary  objectives  ot  the  literature  se.irch  were  to  identify  and 
classify  hardware  and  software  reliability  f.ictors  and  to  identify  existing 
models  and  evaluate  their  applicability  with  respect  to  overall  study 
object ives . 

2.2  Approach 

The  four  major  document  sources  included  in  the  literature  searcli  data 
base  were  the  Rome  Air  Development  Center  customer  files,  National  Techni¬ 
cal  Information  Services,  Martin  Marietta  (Orlando  Aerospace)  Technical 
Information  Cetjter,  and  internal  technical  files.  More  than  500  abstracts 
were  screened  for  the'  literature  search. 

Five  team  members  were  assigned  different  areas  of  responsibility  in 
the  literature  search  ai\d  a  set  of  articles  to  be  covered.  As  material  was 
reviewed,  it  was  entered  into  a  data  base  with  the  reviewer's  annotations. 
The  data  base  constructed  during  this  phase  was  subsequently  used  as  a  ref¬ 
erence  and  source  of  information  throughout  the  performance  of  the  study. 

Over  100  of  the  screened  documents  were  reviewed  in  depth  for  data 
pertinent  to  the  study.  These  were  abstracted  and  commented  upon;  the 
results  were  entered  in  the  study  data  base.  Appendix  B  contains  tl>e 
title,  author,  date,  source,  and  abstract  of  each  document  reviewed. 

2.3  Re  su  1 1  s 

2.3.1  Reliability  Modeling  Approaches 

In  the  course  of  the  literature  search,  software  reliability  models  of 
three  different  categories  were  examined;  measurement  models,  estimation 
models,  and  prediction  models.  One  of  the  primary  findings  of  the  litera¬ 
ture  search  was  that  the  vast  majority  of  software  reliability  effort  has 
historically  been  expended  on  creating  methods  for  measuring  current 
software  reliability  and  extrapolating  that  data  to  estimate  the  future 
reliability  of  existing  software  packages. 

Two  measurement  models,  the  Hecht  Measurement  model  and  the  Nelson 
model,  as  well  as  19  estimation  models  were  examined.  The  majority  of 
estimation  models  are  modifications  of  either  the  Moranda  model  or  the 
Markov  model.  References  to  three  prediction  models  were  found:  the  Motley 
and  Brooks  model;  the  McCall,  Richards,  and  Walters  model;  and  the  Halstead 
model.  Very  little  was  found  concerning  the  prediction  of  a  software 
package  reliability  where  no  historical  failure/debugging  and  run-time  data 
exist  from  which  to  estimate  and  extrapolate.  The  reliability  models 
reviewed  during  the  literature  search  are  listed  in  Table  1. 


TABLE  1.  Software  Reliability  Models. 


Reliability  Measurement  Models 

-  Hecht  measurement  model 

-  Nelson  model 

Reliability  Estimation  Models 

-  Reliability  growth  model  (La  Padjla) 

-  Mills  model 

-  Rudner  model 

-  Jelinski-Moranda  de-eutrophication  model 

-  Jelinski-Moranda  geometric  de-eutrophication  model 

-  Jelinski-Moranda  geometric  poisson  model 

-  Schick-Wolverton  model 

-  Modified  Schick-Wolverton  model 

-  Shooman  exponential  model 

-  Weibull  (Wagoner)  Model 

-  Goel-Okumoto  Bayesian  model 

-  Littlewood-Verrall  Bayesian  model 

-  Shooman-Natara jan  model 

-  Shooman-Tr ivedy  Markov  model 

-  Shooman  Micromodel 

-  Littlewood  Markov  model 

-  Littlewood  semi-Markov  model 

-  Moranda  a  priori  model 

-  Hecht  estimation  model 
••  MUSA  model 

Reliability  Prediction  Models 

-  Motley  and  Brooks  model 

-  McCall,  Richards,  and  Walters  model 

-  Halstead  model 
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The  concensus  of  most  software  reliability  experts  is  that  it  is 
necessary  to  estimate  the  number  of  flaws  remaining  in  a  software  package 
to  extrapolate  or  predict  its  reliability  at  a  given  point  in  the  future. 

Also  evident  is  that  the  vast  majority  of  experts  in  this  field  do  not 
believe  that  software  has  a  constant  failure  rate  analagous  to  the  failure 
rate  of  hardware  represented  by  the  bottom  of  the  bathtub  curve.  The 
opinion  is  held  that  the  software  failure  rate  continues  to  decline 
asymptotically  toward  zero. 

2.3.2  Hardware  Reliability  Factors 

Hardware  reliability  prediction  techniques  have  been  developed  and 
refined  over  the  last  25  years.  Current  hardware  techniques  have  been 
proven  to  be  very  effective  as  reliability  design  tools  and  for  evaluating 
alternative  design  approaches  against  quantitative  reliability  requirements. 
The  factors  which  impact  hardware  reliability  are  well  known  to  reliability 
engineers. 

Hardware  reliability  is  directly  dependent  on  many  factors,  but  the 
major  categories  are: 

_1_  ENVIRONMENT  -  Stresses  imposed  by  the  environment  in  which  the 
hardware  is  to  function 

2  QUALITY  -  Measure  of  the  relative  goodness  of  a  given  part  in  com¬ 
parison  to  similar  parts 

_3  STRESS  RATIO  -  Ratio  of  applied  electrical  stress  to  the  rated 

stress  (voltage  for  capacitors,  power  for  resistors 
and  transistors,  and  current  for  diodes) 

^  TEMPERATURE  -  Ambient  temperature  that  the  part  is  exposed  to  while 
in  an  operational  mode 

TECHNOLOGY  -  Generic  part  type. 

The  most  widely  used  vehicle  for  predicting  electronic  hardware  reli¬ 
ability  is  MIL-HDBK-21 7D.  Some  attributes  may  affect  a  given  part  type 
considerably  more  than  another  part  type.  A  separate  failure  rate  model  is 
supplied  for  each  generic  part-type  to  account  for  these  differences. 

Table  2  illustrates  the  extensive  characterization  of  the  PRODUCT  which  is 
required  to  predict  hardware  reliability.  In  contrast,  software  reliability 
can  be  seen  to  be  much  more  PROCESS  dependent.  Although  software  factors 
such  as  complexity,  number  of  interfaces  and  size  might  indicate  various 
degrees  of  inherent  error  proneness,  it  is  the  process  which  ultimately 
drives  the  number  of  errors  which  remain  in  software.  Other  than  the 
stresses  imposed  by  the  input  domain,  software  is  obviously  not  subject  to 
the  degradation  effects  of  environment  and  aging. 
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TABLE  2.  Factors  for  Part  Failure  Rate  Models 
Discrete  Semiconductors  and  Capacitors. 


Factor  Description 

Common  Factors  -  Used  in  all  or  many  part  categories 

E  Environment  -  Accounts  for  influence  of 

environmental  factors  related  to 
application  categories 

Q  Quality  -  Accounts  for  effects  of 

different  quality  levels 


Discrete  Semiconductors 


A 


R 

C 

S2 


F 


Application  -  Accounts  for  effect  of 
application  in  terms  of  circuit 
function 

Rating  -  Accounts  for  effect  of  maximum 
power  or  current  rating 

Complexity  -  Accounts  for  effect  of 
multiple  devices  in  a  single  package 

Voltage  stress  -  Adjusts  model  for  a 
second  electrical  stress  (application 
voltage)  in  addition  to  wattage 

Frequency  and  peak  operating  power 
factor 


T  Temperature  -  Accounts  for  effects  of 

temperature 

M  Matching  networks  -  Accounts  for  effects 

of  type  of  matching  networks 
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TABLE  2.  Factors  for  Part  Failure  Rate  Models 
Discrete  Semiconductors  and  Capacitors  (Cont). 


Factor 

Capacitors 

SR 


CV 


C 


CF 


Description 


Series  resistance  -  Adjusts  model  for 
the  effect  of  series  resistance  in 
circuit  application  of  some  electro¬ 
lytic  capacitors 

Capacitance  value  -  Adjusts  model  for 
effect  of  capacitance  related  to  case 
size 

Construction  factor  -  accounts  for 
effects  of  hermetic  and  nonherraetic 
seals  on  CL  and  CLR  capacitors 

Configuration  factor  -  Accounts  for 
effects  of  fixed  and  variable  con¬ 
structions  on  CG  capacitors 
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2.3.3  Software  Reliability  Factors 

The  literature  search  revealed  an  intense  interest  within  the  industry 
to  identify  software  characteristics  which  impact  software  reliability. 

In  general,  the  factors  identified  during  the  literature  search  could  be 
categorized  according  to  their  pertinence  to  a  particular  phase  of  the 
overall  software  life  cycle.  Specifically,  the  natural  categorization 
suggested  by  the  literature  search  was  primarily  chronological: 

o  Software  Product  Development  ~  These  factors  are  introduced  during 
the  development  phase.  They  are  essentially  design  and  code  tech¬ 
niques. 

o  Software  Product  Functional  Performance  -  These  factors  measure  the 
software's  response  to  the  requirements  specification. 

o  Software  Initial  Operation  -  These  factors  relate  to  the  usability 
of  the  software  product. 

o  Software  Product  Revision  -  Most  software  is  expanded  or  otherwise 
altered  during  its  useful  life.  These  factors  influence  the  reli¬ 
ability  of  the  modified  product. 

o  Software  Transition  -  These  factors  address  the  characteristics  of 
software  that  allow  it  to  adjust  to  changing  hardware  environments. 

The  remainder  of  this  section  describes  the  software  characteristics 
identified  during  the  literature  search.  Although  they  are  presented  as 
categorized  above,  it  should  be  noted  that  several  re-categorizations  were 
performed  before  the  list  could  be  used  as  factors  within  the  methodology. 
First,  the  factors  were  re-categorized  into  inherent,  design/development 
and  application  characteristics  (see  Section  3.0).  Next  they  were  priori¬ 
tized  in  accordance  with  their  relative  influences  on  system  reliability 
(Section  4.3.1).  Finally,  the  most  significant  factors  were  re-categorized 
again  (Section  4.3.2),  this  time  to  separate  the  inherent  characteristics 
of  the  software  PRODUCT  from  the  error  avoidance  and  error  detection 
characteristics  of  the  software  development  PROCESS. 

Software  Product  Development 

Software  reliability  will  only  be  achieved  if  it  is  a  requirement  that 
IS  contractually  specified  and  subsequently  designed  and  coded  into  the 
software  product  during  the  development  phase  of  the  life  cycle. 

Virtually  all  sources  studied  during  the  literature  search  concluded 
that  reliable  software  is  achievable  only  by  the  application  of  a  sound 
engineering  approach.  Although  different  authors  favor  different  tech¬ 
niques  and  methods  for  developing  reliable  software,  most  converge  on 
factors  that: 


Facilitate  early  recognition  of  problems 

^  Increase  the  probability  of  fault  detection 

_3  Simplify  fault  isolation 

^  Isolate  functional  and  logical  activities. 

The  characteristics  or  factors  most  commonly  addressed  recognize  and 
support  the  current  state-of-the-art  software  development  techniques.  In 
the  following  list,  complexity  is  the  single  most-important  factor  relating 
to  software  reliability,  while  all  others  are  agreed-upon  methods  of  reduc¬ 
ing  it: 

1  Complexity  -  This  is  a  negative  factor  that  tends  to  make  computer 
programs  incomprehensible.  It  includes  much  more  than  other  terms 
such  as  "readability"  or  "modularity."  Complexity  can  be  introduced 
into  requirements,  design,  code,  and  even  test  activities.  To 
avoid,  detect,  and  correct  faults  prior  to  the  operational  phase 
when  reliability  will  be  needed,  the  software  must  be  reviewed, 
discussed,  inspected,  and  tested  by  human  beings  of  limited  capacity 
to  recognize  very  complex  interrelationships.  Complexity  seriously 
impedes  and  precludes  these  activities. 

^  Top-down  Functional  Decomposition  -  In  one  way  or  another,  most 

authors  suggested  functional  decomposition  or  functional  threading 
to  minimize  complexity. 

_3  Modularization  -  The  segmentation  of  computer  programs  into  single¬ 
purpose,  single-entry,  single-exit  modules  was  the  most  popular 
technique  discussed  in  the  literature. 

^  Hierarchical  Design  -  Most  authors  acknowledged  the  significant 
reduction  in  complexity  that  can  be  achieved  by  enforcing  a  hier¬ 
archical  structure  of  module  segment  calling  and  controlling 
relat ionships . 

^  Structured  Approaches  -  The  advantages  of  structured  techniques  are 
well  known.  Their  impact  on  software  complexity  is  clear,  but  as 
of  now,  unquant i f iab le . 

Software  Product  Functional  Performance 

The  factors  discussed  in  this  section  relate  to  the  degree  to  which  a 
software  product  meets  its  stated  performance  requirements.  Results  of  the 
literature  search  indicate  a  consensus  that  the  following  software  factors 
are  signif leant : 

Correctness  -  Ability  of  the  computer  program  to  accurately  perform 
all  of  the  functions  required  by  the  specifications. 


2  ValidiLy  -  Ability  of  the  computer  program  to  provide  the  perform¬ 
ance,  functions,  and  interfaces  that  are  sufficient  for  beneficial 
application  in  the  intended  user  environment.  The  distinction 
between  this  and  the  definition  of  "correctness"  should  be  noted. 
Whereas  correctness  checks  for  accomplishment  of  the  specified 
objectives,  validity  pertains  to  specifications  as  well  as  result¬ 
ing  software. 

_3  Generality  -  Ability  of  the  computer  program  to  perform  its  intend¬ 
ed  functions  over  a  wide  range  of  usage  modes  and  inputs,  even  when 
a  range  is  not  directly  specified  as  a  requirement. 

^  Testability  -  Characteristic  of  a  computer  program  that  allows  its 
functional  requirements  to  be  logically  separated  to  allow  step-by- 
step  testing  of  each  aspect  of  the  program. 

5  Efficiency  -  Measure  of  the  use  of  high-performance  algorithms  and 
conservative  use  of  resources  to  minimize  the  cost  of  computer  pro¬ 
gram  operation. 

Software  Initial  Operation 

These  factors  influence  the  performance  of  the  software  d:  ring  the 
operational  phase.  In  this  section,  the  phrase  "initial  operati.n"  is  used 
to  isolate  those  characteristics  of  software  that  pertain  only  to  the 
original  computer  program  that  was  specified  and  developed.  It  does  not 
include  those  aspects  that  relate  to  the  software's  ability  to  be  maintain¬ 
ed.  Those  aspects  will  be  covered  elsewhere.  The  following  factors  were 
identified  in  the  literature  as  being  pertinent  to  the  initial  operation  of 
a  computer  program: 

_1_  Usability  -  Characteristic  of  software  that  is  indicative  of  its 
responsiveness  to  human  factors  considerations.  It  is  a  measure  of 
how  well  it  has  used  natural  and  convenient  techniques  for  human 
operat ion . 

^  Resilience  -  Sometimes  referred  to  as  ROBUSTNESS  or  in  hardware 

terminology,  SENSITIVITY;  this  is  a  measure  of  a  computer  program's 
ability  to  perform  in  a  reasonable  manner,  despite  violations  of 
assumed  usage  and  input  conventions. 

3  Fault  Tolerance  -  Ability  of  the  computer  program  to  perform 
correctly,  despite  the  presence  of  error  conditions. 

Software  Product  Revision 


Software  reliability  is  concerned  with  the  usage  of  computer  programs 
over  some  operational  life  in  a  specified  operational  environment.  The 
existence  of  totally  unmodified  software  in  the  operational  environment  is 
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a  rarity  if,  in  fact,  any  exist.  The  literature  search  revealed  several 
pertinent  factors: 

J_  Clarity  -  Ability  of  the  computer  program  to  be  easily  understood. 
It  is  a  measure,  not  only  of  the  computer  program  itself,  but  also 
of  its  supporting  documentation. 

2^  Readability  -  Measure  of  how  well  a  skilled  programmer,  not  the 
original  creator,  can  understand  the  program  and  correlate  it  to 
the  original  and  new  requirements. 

3  Maintainability  -  Catch-all  term  used  to  summarize  all  the  features 
of  a  computer  program  that  allow  it  to  be  easily  altered  or 
expanded.  It  considers  all  of  the  other  factors  shown  in  this 
list. 


^  Modifiability  -  Measurement  that  includes  consideration  of  the 

extent  to  which  likely  candidates  for  change  are  isolated  from  the 
rest  of  a  computer  program.  As  an  example,  the  isolation  of  input 
and  output  routines  that  are  hardware-  or  human-dependent  would 
increase  the  program's  modifiability  since  these  are  very  vulner¬ 
able  areas  to  post-delivery  modifications. 

_5  Flexibility  -  Measure  of  the  computer  program's  design,  which 

allows  it  to  perform  or  to  be  easily  modified  to  perform  functions 
beyond  the  scope  of  its  original  requirements. 

Software  Transition 

Software  transition  is  the  process  of  changing  the  environment  of  a 
software  product  either  by  installing  it  in  another  system,  in  another 
application,  or  in  another  software  product.  Similar  to  hardware, 
thoroughly  tested  software  can  be  regarded  as  a  standard  part  that  can  be 
incorporated  into  another  application. 

As  software  is  better  understood  by  nonsoftware  personnel,  it  is  be¬ 
coming  evident  that  many  standard  hardware  techniques  and  methods  can  be 
applied  to  Its  usage.  One  idea  that  seems  to  be  gaining  in  popularity  is 
that  of  the  development  and  use  of  general  purpose  software  that  has  been 
identified  as  having  high  reliability.  This  idea  is  similar  to  the  usage 
of  Hi-Rel  parts  in  the  development  of  new  hardware  systems.  Some  of  the 
software  factors  discovered  during  the  literature  search  that  seem  to 
indicate  the  degree  of  software  transition  reliability  are  listed  below: 


_1_  Portability  -  Characteristic  of  computer  software  that  allows  it  to 
be  used  in  a  computer  environment  different  from  the  one  for  which 
it  was  originally  designed.  Use  of  standard  high-level  languages 
is  one  of  the  ways  to  increase  portability. 
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2  Reusability  -  Measure  of  the  extent  to  which  a  computer  program  can 
be  used  in  a  different  application  from  the  one  for  which  it  wcs 
developed . 

3  Interoperability  -  Measure  of  the  ease  by  which  a  computer  program 
can  be  made  to  interface  with  other  computer  programs. 

2.4  Conclusions 

The  literature  search,  while  providing  extensive  background  informa¬ 
tion  on  research  already  accomplished,  revealed  that  no  existing  model  can 
be  directly  applied  to  the  problem  of  reliability  prediction  prior  to 
development.  It  was  concluded  that  the  methodology  required  would  have  to 
be  developed  during  this  study  and  would  require  the  usage  of  parameters 
available  to  the  reliability  engineer  at  the  beginning  of  the  development 
cycle . 

It  was  also  concluded  that  the  natural  (chronological)  categorization 
evidenced  by  the  literature  search  did  not  adequately  separate  software 
product  characteristics  from  software  development  process  characteristics. 
Many  of  the  factors  identified  are,  in  fact,  measures  or  qualities  of  the 
resulting  software  product,  not  determinants  of  its  reliability. 
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3.0  CHARACTERIZATION  TECHNIQUES  FOR  RELIABILITY  PREDICTION 

3.1  Overview  and  Objectives 

Concurrent  with  the  literature  search  discussed  in  Section  2.0,  a  pre¬ 
liminary  methodology  was  developed.  The  objective  of  this  phase  of  the 
study  was  to  reevaluate  the  factors  and  characteristics  investigated  during 
the  literature  search  and  to  characterize  them  in  a  manner  consistent  with 
the  methodology. 

3.2  Approach 

The  approach  used  during  this  phase  was  primarily  analytical.  First, 
the  overall  system  and  mission  reliability  characteristics  were  investigated 
to  establish  the  rationale  and  assumptions  of  the  model.  Secondly,  the 
software  characteristics  identified  during  the  literature  search  were  re¬ 
evaluated  in  light  ol  the  anticipated  methodology  which  recognizes  the 
significant  relationship  between  the  reliability  of  the  software  product 
and  the  process  which  creates  it.  The  factors  already  presented  in  Section 

2.3.3  were  simultaneously  purged,  expanded,  and  re-categorized  (Table  3)  to 
facilitate  their  use  in  the  model. 

3.3  Re  su 1 1  s 

Reliability  prediction  of  total  systems,  which  includes  the  effects  of 
embedded  software,  must  simultaneously  consider  the  effects  of  both  hard¬ 
ware  and  software.  The  analysis  performed  during  this  phase  verified  that, 
although  they  are  closely  related,  hardware  and  software  can  be  evaluated 
independently  for  purposes  of  reliability  prediction.  Furthermore,  the 
analysis  concluded  that  mission  reliability  prediction  should  be  the  pri¬ 
mary  goal  of  the  methodology. 

3.3.1  System  Reliability 

System  reliability,  as  previously  defined,  is  the  probability  that  the 
required  system  will  perform  its  intended  functions  for  prescribed  missions 
and  time  periods  in  the  specified  operating  environment.  Before  describing 
the  details  of  Llie  approach,  it  is  necessary  to  eliminate  the  ambiguities 
of  itie  detinition  itself.  Furthermore,  it  is  necessary  to  constrain  the 
problem  definition  to  a  mon?  solvable  and  practical  one. 

Ml  SSI  on  Re  1 lab 1 1  it  y 


First  and  loremost,  it  should  be  recognized  that  although  a  given 
system  may  have  various  operational  missions,  the  motivation  tor  performing 
a  reliability  pr</il  ict  ion  is  to  obtain  a  quantitative  measure  of  the  likeli¬ 
hood  of  success  of  a  specilic  mission.  Although  the  prediction  and  analy¬ 
sis  of  availability,  maintainability,  and  supportab i 1  it y  require  considera¬ 
tion  of  all  possible  uses  of  a  given  system,  the  methodology  presented 
herein  is  directed  toward  the  prediction  of  system  reliability  for  a 
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TABLE  3.  Software  Characteristics. 
Operational  Requirements 

-  Predominantly  control 

-  Predominantly  computational 

-  Predominantly  input/output 

-  Predominantly  real-time 

-  Predominantly  interactive 

Environmental  Requirements 

-  Number  of  hardware  interfaces 

-  Number  of  software  interfaces 

-  Number  of  human  interfaces 

Size  Considerations 

-  Number  of  functions  performed 

-  Overall  program  size 

-  Number  of  compilation  units 

-  Maximum  size  per  unit 

Complexity  Considerat ions 

-  Number  of  entries  and  exits 

-  Number  of  control  variables 

-  Use  of  single-function  modules 

-  Number  of  modules 

-  Maximum  module  size 

-  Hierarchical  control  between  modules 

-  Logical  coupling  between  modules 

-  Data  coupling  between  modules 

Organizational  Considerations 

-  Separate  design  and  coding 

-  Independent  test  organization 

-  Independent  quality  assurance 

-  Independent  configuration  control 

-  Independent  verification/validation 

-  Programming  team  structure 

-  Educational  level  of  team  members 

-  Experience  level  of  team  members 
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TABLE  3.  Software  Characteristics  (Cont) 
Methods  Used 

-  Definition/enforcement  of  standards 

-  Use  of  High  Order  Language  (HOL) 

-  Formal  reviews  (PDR,  CDR,  etc.) 

-  Frequent  walkthroughs 

-  Top-down  and  structured  approaches 

-  Unit  development  folders 

-  Software  development  library 

-  Formal  change  and  error  reporting 

-  Progress  and  status  reporting 

Design  Approach 

-  Modular  construction 

-  Structured  design 

-  Structured  code 

Tools  Used 

-  Flow  charts 

-  Structure  charts 

-  Decision  tables 

-  HIPO  charts 

Documentation 

-  System  requirements  specification 

-  Software  requirements  specification 

-  Interface  design  specification 

-  Software  design  specification 

-  Source  listings 

-  Test  plans,  procedures  and  reports 

-  S/W  development  plan 

-  S/W  quality  assurance  plan 

-  S/W  configuration  management  plan 

-  Requirements  traceability  matrix 

-  Version  description  document 

-  Software  discrepancy  reports 

Duty  Cycle 

-  Constant  mission  usage 

-  Periodic  mission  usage 

-  Infrequent  mission  usage 
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TABLE  3.  Software  Characteristics  (Cont). 
Environment 

-  Variability  of  hardware 

-  Training  level  of  operators 

-  Variability  of  input  data 

-  Variability  of  outputs 

-  Degree  of  human  interaction 

Non-operational  Usage 

-  Training  exercises 

-  Periodic  self  test 

-  Built-in  diagnostics 

Modification/Error  Correction 


-  Performed  in  the  field 

-  Performed  at  depot 

-  Performed  at  factory 

Qualitative  Characteristics 


-  Correctness 

-  Validity 

-  Generality 

-  Testability 

-  Efficiency/economy 

-  Resilience  (Robustness) 

-  Usability 

-  Fault  tolerance 

-  Clarity 

-  Readability 

-  Maintainability 

-  Modifiability 

-  Flexibility 

-  Portability 

-  Reusability 

-  Interoperability 


specific  mission.  While  the  prediction  of  multimission  system  reliability 
is  not  specifically  addressed,  it  should  be  noted  that  the  determination  of 
single-mission  system  reliabilities  is  the  primary  and  most  critical 
element  of  multimission  predictions.  Where  such  predictions  are  required, 
the  method  presented  could  be  applied  to  each  defined  mission;  these 
results  are  combined  mathematically  using  classical  statistical  methods. 

System  Definition 

Like  most  things  in  nature,  knowledge  of  a  system  can  be  obtained 
through  a  detailed  understanding  of  its  composition.  Specifically, 
knowledge  of  component  characteristics  that  comprise  the  system  and  their 
interactions  with  each  other  is  equivalent  to  knowledge  of  the  system. 


In  the  case  of  computerized  systems,  the  initial  decomposition  con¬ 
sists  of  those  components  that  may  be  categorized  as  hardware,  software,  or 
human.  Of  these,  software  is  the  least  understood  quantitatively.  Hence, 
the  methodology  developed  concentrates  on  the  contributions  of  software  to 
the  overall  reliability  of  the  system.  It  is  assumed  that  the  prediction 
and  measurement  techniques  currently  being  applied  to  hardware  systems  are 
adequate  for  the  description  of  Che  hardware  contributions  to  system  reli¬ 
ability.  Furthermore,  it  is  assumed  tnat  human  participation  in  Che  mis¬ 
sion  accomplishment  is  limited  Co  controlling  the  hardware/software 
components  by  providing  input  data  or  reacting  to  outputs  as  necessary.  As 
such,  Che  human  component  can  be  considered  as  part  of  the  operational 
environment,  not  part  of  the  system.  This  does  not  impose  a  limitation  on 
the  methodology,  but  rather  is  a  logical  way  of  representing  an  automated 
system.  Therefore,  Che  methodology  considers  a  system  Co  be  comprised  of 
hardware  and  software  components. 


Interactions  of  System  Components 


Knowledge  of  the  interaction  of  system  components  with  one  another  is 
critical  to  the  understanding  of  the  system  itself.  There  are  two  major 
categories  of  interaction  chat  must  be  addressed.  The  first  involves  in¬ 
formation  exchange  that  provides  necessary  inputs  to  a  component.  The 
second  involves  behavior  modification,  whereby  one  component  influences  the 
manner  in  which  another  accomplishes  its  allocated  requirements.  The  dis¬ 
tinction  can  be  made  by  considering  that  Che  first  category  allows  one 
component  to  alter  the  environment  of  Che  other  by  presenting  new  conditions 
tor  It  to  operate  upon,  while  the  second  category  allows  a  component  to 
actually  alter  the  other  component.  This  is  purely  an  abstraction  and  does 
not  represent  the  physical  modification  of  system  components,  but  instead, 
a  logical  modification  of  the  system  definition.  For  example,  the  exchange 
of  commands  and/or  data  between  the  hardware  and  software  components  of  a 
system  tall  entirely  in  the  first  category,  even  when  the  exchange  is 
entirely  incorrect.  The  ability  of  a  component  to  perform  its  intended 
functions  despite  erroneous  inputs  is  usually  referred  to  as  its  robustness, 
and  IS  somewhat  measurable.  On  the  other  nand,  fault-tolerance  imp’’es 
that  a  fault  has  occurred  and  that  the  system  will  self-adjust  Co  meet  its 


requirements.  That  is,  the  system  is  automatically  altered  to  a  new  con- 
f igurat ion. 


Although  considerable  knowledge  is  known  about  active  and  dormant 
liardware  redundancy  and  mathematical  techniques  have  been  devised  for 
measuring  its  effects,  this  knowledge  is  not  directly  transferable  to  soft¬ 
ware  components.  Identical  hardware  components  can  be  redundantly  config¬ 
ured  to  increase  reliability,  but  identical  software  components  will  react 
identically  to  the  same  environment.  Redundant  software  must  be  different 
software,  and  the  automatic  replacement  of  a  portion  of  software  when  it 
fails  is  tantamount  to  a  redefinition  of  the  system.  The  situation  where 
the  occurrence  of  a  hardware  failure  causes  a  software  compensation  (or 
vice  versa)  adds  a  degree  of  complexity,  which  is  currently  beyond  the 
state  of  the  art  of  practical  measurement.  Extensive  work  is  currently 
under  way  with  respect  to  fault  tolerance,  both  on  a  subcomponent  as  well 
as  a  system  level.  There  are  many  respected  system  designers  and  analysts 
who  feel  that  fault  tolerance  is  the  ultimate  answer  to  system  reliability 
problems.  The  methodology  does  not  directly  incorporate  software  fault 
tolerance  consrderat ions  but  does  support  such  analyses.  First,  the  Markov 
technique  allows  the  analyst  to  incorporate  redundant  software  into  the 
overall  model  in  the  same  manner  as  the  other  software  components.  The 
path  probability  of  the  redundant  software  is  simply  the  probability  that 
the  primary  software  component  will  fail.  Secondly,  when  the  model  is 
exercised,  sensitivity  information,  in  the  form  of  relative  utilization,  is 
produced  for  each  software  component.  The  analyst  can,  therefore,  use  the 
model  to  identify  candidates  for  application  of  fault  tolerance  techniques. 


Conclusions  Concerning  Svstem  Reliabilit 


An  automated  system  can  be  described  by  two  major  subsystems,  its 
hardware  and  its  software,  and  that  for  a  given  system  configuration,  the 
interaction  between  these  components  is  limited  to  information  exchange. 

We  can,  therefore,  analyze  them  independently  by  considering  the  hardware 
interface  as  part  of  the  software's  operating  environment  and  vice  versa. 

By  restricting  our  analysis  to  mission-critical  hardware  and  software,  we 
can  clearly  see  that  for  a  particular  mission,  a  combined  hardware/ software 
system  can  be  represented  by  the  serial  configuration  of  a  hardware  and 
software  component  and  that  the  system  reliability  is  simply  the  product  of 
the  two  component  reliabilities  as  determined  from  their  respective  con¬ 
figurations  and  environments. 

3.3.2  Hardware  Subsystem  Reliability 

Both  the  literature  search  and  the  preceding  analysis  confirmed  that 
hardware  and  software  reliability  prediction  could  be  accomplished  in¬ 
dependently.  Therefore,  it  was  decided  that  the  methodology  would  concen¬ 
trate  exclusively  on  software  reliability  prediction.  System  hardware 
reliability  prediction  is  adequately  accomplished  via  M IL-HUBK-2 1 7D . 


3.3.3  Soil  ware  Subsystem  Kel lability 

Unlike  hardware,  soltware  lailures  are  not  physical  but  rather  logical 
in  nature.  Soltware  does  not  degrade  with  age,  nor  does  it  degrade  due  to 
the  physical  stresses  usually  considered  in  hardware  reliability  predic¬ 
tion.  The  reliability  of  the  software  is  the  direct  result  of  the  process 
used  to  produce  it.  If  the  development  process  allows  errors  to  be  made 
and  go  undetected,  the  software  product  will  eventually  fail.  During  this 
phase  of  the  study,  the  data  collected  during  the  literature  search  was 
eKpanded  and  re-categorized  in  order  to  isolate  those  characteristics  which 
affected  the  quality,  albeit  reliability,  of  the  development  process.  The 
overall  intent  was  to  eventually  identify  intrinsic  or  inherent  factors 
which  influence  the  difficulty  of  producing  software  and  to  identify 
development  and  design  techniques  which  may  be  applied  to  increase  the 
liklihood  of  avoiding  and  detecting  errors. 

Table  3  presented  the  expanded  list  of  software  factors  which  was 
created.  The  list  was  constructed  specifically  for  use  as  a  tool  for 
further  data  collecting  and  as  a  format  for  the  Delphi  survey  performed 
during  the  next  phase  of  the  study.  Since  one  of  the  objectives  of  the 
survey  was  to  rank  the  effects  of  individual  factors,  they  were  presented 
in  subgroups  as  shown  in  the  table.  The  subgroup  titles  are 
self-explanatory.  They  were  chosen  as  potential  categories  from  which  to 
determine  pi  factors  for  use  in  the  model.  Although  this  categorization 
scheme  was,  once  again,  revised  as  a  result  of  the  first  survey,  it  formed 
the  basis  for  the  final  form  used  within  the  methodology. 

Section  4.0  of  this  report  discusses  the  survey  in  detail.  The  survey 
forms,  instructions,  and  results  are  included  in  the  appendices.  Section 
5.0  presents  the  prediction  methodology,  and  the  manner  in  which  the  soft¬ 
ware  factors  are  incorporated. 

3.4  Conclusions 

The  priniaiy  conclusion  from  this  phase  of  Ihe  study  was  tliat  soft¬ 
ware  reliability  pr<*dicCion  could  be  accomplished  in  a  manner  similar  to 
classical  hardware  techniques.  That  is,  the  overall  software  reliability 
prediction  can  be  accomplished  by  decomposing  the  software  into  its 
component  parts,  predicting  the  reliability  of  each  component,  and  math¬ 
ematically  combining  the  individual  predictions.  It  was  also  concluded 
that,  whereas  har<lwarv-  ri'liability  prediction  is  primarily  dependent  on 
PRODUCT  cha rac t e r i St i c s  and  physical  construction,  software  reliability 
prediction  is  heavily  influenced  by  PROCESS  cha rac I  i- r i st ic s  and  mission- 
dependent  path  probabilities.  It  was  also  determined  that  too  many 
factors  were  being  considered  and  that  the  pilot  survi^y  should  be  used  to 
identify  those  factors  most  critical  to  the  methodology. 


4.0  TWO-PASS  SURVEY 


4.1  Overview  and  Objectives 

As  a  result  of  the  literature  search  and  the  subsequent  characteriza¬ 
tion  phase,  an  extensive  list  of  software  factors  and  attributes  was  com¬ 
piled.  The  surveys  were  conducted  to  identify  the  most  significant  factors 
and  to  quantify  their  effects  on  reliability.  The  preferred  approach  was 
to  quantify  the  factors  based  on  historical  software  error  data.  It  was 
found,  however,  that  it  was  not  possible  to  derive  the  required  quantitative 
information  from  currently  available  data.  Specialized  data  collection  and 
experimentation  efforts  are  required  to  fully  quantify  Che  factor  effects. 

4.2  Approach 

The  approach  taken  was  to  divide  the  survey  into  two  phases.  First,  a 
pilot  survey  was  accomplished  to  identify  Che  factors  most  significant  Co 
reliability  prediction.  Second,  a  full-scale  survey  was  conducted  to  quan¬ 
tify  Che  effects  of  Chose  factors.  Since  Che  phases  of  the  survey  were 
performed  sequentially,  the  remainder  of  this  section  is  outlined  to  separ¬ 
ate  and  highlight  the  objectives,  approaches  and  results  of  each  of  the 
surveys . 

4.3  Results 

4.3.1  Pilot  Survey 
Object ives 

The  literature  search  produced  an  abundance  of  characteristics  and 
development  techniques  Chat  relate  to  software  reliability  (Table  3).  To 
learn  more  about  these  factors,  a  pilot  survey  was  conducted.  The  objec¬ 
tive  of  the  survey  was  to  obtain  a  qualitative  ranking  of  the  software 
factors,  characteristics,  and  techniques  in  terms  of  their  impact  on 
software  reliability.  An  implied  objective  was  the  elimination,  for 
further  consideration,  of  those  factors  perceived  as  having  little  or  no 
impact . 

Approach 

A  survey  lotm  was  devised  using  the  factors  and  attributes  identified 
during  the  literature  search.  They  were  categorized  into  the  groups 
already  described  in  Section  3.0.  The  survey  form  was  actually  a  matrix 
with  the  factors  forming  Che  rows  of  the  matrix.  Eight  columns  were 
included.  The  first  was  titled  "System  Reliability  Impact,"  and  the  other 
seven  were  labeled  to  correspond  to  the  major  software  error  categories: 

-  Specification  Errors 

-  Design  Errors 


-  Coding  Errors 

-  Software  Interface  Errors 

-  Hardware  Interface  Errors 

-  Human  Interface  Errors 

-  Capacity  Problems. 

Participants  were  asked  to  rate  the  degree  of  correlation  between  each  of 
the  software  factors  listed  in  the  rows  of  the  matrix  and  each  impact  area 
listed  in  the  columns.  The  four  possible  ratings  were  H  (high),  M  (medium), 
L  (low),  and  0  (zero)  correlation.  It  is  important  to  note  that  the  par¬ 
ticipants  were  asked  simply  to  indicate  a  correlation  level  and  not  a 
positive  or  negative  relationship.  The  goal  was  to  identify  those  factors 
that  have  a  significant  effect  on  reliability  —  not  to  judge  whether  the 
effect  was  good  or  bad.  The  pilot  survey  form  and  its  instructions  are 
included  in  Appendix  C. 

Result  s 

The  survey  was  distributed  to  30  hand-picked  experts  who  were  selected 
for  their  known  interest  and  expertise  in  software  development.  Twenty- 
three  responded.  Some  confusion  resulted  from  the  decision  to  design  the 
form  without  provisions  to  distinguish  positive  and  negative  effects.  Many 
participants  were  uneasy  with  assigning  the  same  rating  to  a  factor  that 
they  felt  had  a  high  positive  influence  on  reliability  and  a  factor  that 
they  felt  had  a  high  negative  influence.  Fortunately,  since  the  distribu¬ 
tion  was  very  limited,  it  was  possible  to  clarify  the  intent  through 
personal  contact. 

The  responses  were  recorded  in  a  computerized  data  base  and  converted 
into  numerical  values  of  nine,  five,  one,  and  zero  for  high,  medium,  low 
and  no  correlation,  respectively.  Averages  were  computed  for  each  factor, 
each  group  of  factors,  and  each  error  category.  They  were  then  ranked 
numerically  by  averages  within  each  category  and  by  overall  average. 

Appendix  D  presents  the  results  of  the  analysis  performed.  Of  the 
factors  themselves,  interface  and  requirement  specifications  were  rated  as 
having  the  greatest  impact  on  system  reliability.  Many  of  the  participants 
remarked  on  the  form  that  complete  and  unambiguous  specifications  have  a 
strong,  positive  effect  on  reliability  while  incomplete  or  incorrect  ones 
have  an  equally  strong,  negative  effect.  Frequent  walkthroughs  were  rated 
as  the  most  influencial  technique,  followed  closely  by  the  definition  and 
use  of  predefined  standards  and  conventions.  Curiously,  the  use  of 
standards  rated  slightly  higher  than  structured  approaches  and  formal 
reviews.  The  inherent  factor  having  the  highest  rating  was  real-time 
applications.  Since  it  is  generally  felt  that  software  required  to  operate 
in  real  time  tends  to  be  more  complex  than  other  applications,  this  result 
was  not  surprising. 

The  ranking  of  the  results  also  indicated  that  many  of  the  factors 
were  not  felt  to  be  significant  to  reliability  prediction.  Most  were 
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obviously  unrelated  but  included  in  the  survey  tor  completeness  and  object¬ 
ivity.  Factors  such  as  economy,  readibility,  flexibility  and  most  of  the 
other  -ilities  were  rated  very  low.  Although  each  is  significant,  they  are 
attributes  of  the  finished  software  product,  not  determinants  of  its  reli¬ 
ability.  Some  other  lowly  rated  factors  were  not  so  obvious.  The  level  of 
training  and  experience  of  the  operational  users  was  not  considered  sig¬ 
nificant.  The  educational  level  of  the  software  developers  was  rated  much 
lower  than  their  experience  level. 

As  a  result  of  the  pilot  survey,  many  factors  were  eliminated  from 
further  analysis  and  those  that  remained  were  rearranged  into  the  categor¬ 
ies  that  would  be  utilized  in  the  methodology.  Specifically,  it  was  noted 
that  software  factors  could  best  be  categorized  into  groups  of  inherent  and 
developmental  characteristics.  The  developmental  characteristics,  in  turn, 
could  best  be  described  in  relation  to  their  ability  to  avoid  errors  or  to 
detect  (and  correct)  errors.  This  categorization  was  reinforced  by  the 
results  of  the  pilot  survey. 

4.3.2  Full-Scale  Survey 

Object ive  s 

The  pilot  survey  revealed  the  characteristics  of  software  that  have  a 
significant  impact  on  reliability.  It  was  necessary,  however,  to  quantify 
these  characteristics  and  factors  in  some  way;  therefore,  a  second,  full- 
scale  survey  was  conducted.  The  objectives  of  the  full-scale  survey  were 
to  obtain  a  quantitative  ranking  of  the  software  factors  and  to  determine 
values  for  the  oarameters  in  the  methodology. 


Using  the  results  of  the  pilot  survey,  the  survey  form  was  redesigned 
to  include  only  the  significant  software  factors  and  to  reflect  the  struc¬ 
ture  of  the  methodology  being  developed.  The  inherent  characteristics  and 
development  techniques  of  software  were  combined  with  development  phases 
and  error  types  to  produce  the  new  questionnaire. 

The  survey  was  sent  to  approximately  350  persons  who  are  involved  with 
software  and/ot  reliability.  To  avoid  ambiguity,  detailed  instructions  and 
definitions  of  all  terms  were  included  with  the  survey. 

The  full-scale  survey  form  and  its  instructions,  consisting  of  five 
pages,  each  different,  are  included  in  Appendix  E.  The  first  sheet  listed 
several  groups  of  inherent  characteristics  of  software.  Participants  were 
asked  to  rank  the  groups  in  order  of  importance  to  system  reliability.  The 
individual  characteristics  were  also  ranked  within  the  groups. 

The  same  inherent  characteristics  were  listed  again  on  Sheets  2  and  3. 
Tnere  wer-  four  columns  <in  Sheet  2,  representing  four  phases  of  development 
The  entry  'or  each  row  and  column  combination  represented  the  relative 
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quantity  of  errors  introduced  by  that  particuiar  factor  during  that  par¬ 
ticular  development  phase.  The  four  columns  on  Sheet  3  represented  four 
types  of  errors.  The  entries  on  that  sheet  indicated  the  percentage  of 
each  type  of  error  present  due  to  each  characteristic. 

The  column  headings  on  Sheets  4  and  5  were  tlie  same  four  error  types 
from  Sheet  3.  Whereas  Sheet  3  was  used  to  quantify  the  distribution  of 
error  types  caused  by  inherent  characteristics,  Sheet  4  listed  development 
techniques  and  was  used  to  quantify  the  effectiveness  of  each  technique  in 
avoiding  each  error  type.  Sheet  5  was  similarly  used  to  quantify  the 
effectiveness  of  error  detection  mechanisms.  The  intent  was  to  quantify 
the  effectiveness  of  the  software  development  process.  This  intent  was 
deliberately  hidden  from  the  participants  to  avoid  individual  bias  and 
pre-supposition.  In  fact,  the  participant  was  asked  to  treat  each  sheet 
independently  from  the  others. 

Unlike  the  pilot  survey,  this  pass  included  a  cover  sheet  to  record 
the  general  educational  and  experience  level  of  the  respondent.  In  addi¬ 
tion,  the  respondent  was  asked  to  identify  his  primary  area  of  interest  and 
general  qualifications.  The  respondent  was  also  invited  to  comment  on  the 
adequacy  and  relevancy  of  the  survey  itself. 

Results 

Approximately  100  responses  were  received.  Each  response  was  recorded 
in  a  computerized  data  base  and  combined  statistically  with  all  other 
responses.  Analysis  of  the  data  included  on  the  cover  sheet  indicated  that 
the  participants  were  highly  qualified  software  professionals.  Seventy- 
seven  percent  work  in  the  software  industry,  18  percent  work  for  government 
agencies,  and  the  rest  are  associated  with  universities.  Seventy-three 
percent  of  those  responding  have  at  least  10  years  experience  in  software 
development,  and  more  than  half  have  a  graduate  degree.  The  detailed  pro¬ 
files  are  presented  in  Appendix  F  as  are  all  other  results  of  the  survey. 

The  rankings  on  Sheet  1  confirmed  the  pilot  survey  results  that  des¬ 
cribed  the  relative  importance  of  the  inherent  characteristics  of  software. 
Modules  that  involve  predominantly  real-time  implementation  are  ranked  as 
most  critical  to  software  reliability.  Control  and  interactive  modules 
were  ranked  second  in  importance.  The  type  of  software  seen  as  having  the 
least  effect  on  reliability  is  a  predominantly  coraputat ional  application. 

According  to  the  survey,  the  number  of  interfaces  is  more  critical  to 
reliability  than  the  type  of  interface  (hardware,  software,  or  human).  The 
complexity  of  the  operations  performed  by  a  module  was  deemed  more  important 
than  the  quantity  of  operations. 

The  most  effective  error  avoidance  mechanism,  according  to  the  survey 
IS  the  rigid  control  of  requirements  and  design  specifications.  Structured 
design  approaches  were  likewise  rated  to  be  highly  effective.  Since  the 
pilot  survey  had  been  used  to  eliminate  less  important  factors,  nearly  all 
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avoidance  mechanisms  listed  were  rated  as  being  inflective.  Surprisingly, 
the  lowest  ranked  avoidance  mechanisms  involved  con  f  ignr.i  t  i  on  control. 

The  results  indicated  that  frequent  walkthroughs  are  the  most 
effective  means  of  detecting  errors,  followed  closely  by  inform.il  unit 
level  testing.  Quality  audits  were  ranked  considerably  less  effect  ivo. 

Most  formal  reviews  were  rated  equally  effective  with  the  Critical  Design 
Review  being  identified  as  having  a  slight  edge  on  error  detection 
capability.  All  of  the  testing  procedures  were  rated  about  the  same. 

4.4  Conclusions 

The  surveys  performed  were  invaluable  to  the  development  ol  tlu’  solt- 
ware  reliability  prediction  methodology.  The  pilot  survey  serveil  to  c'llec- 
tively  isolate  tliose  factors  which  truly  affect  reliability;  tlie  full-scale 
survey  enabled  a  preliminary  quantification  of  the  effects  of  those  f.ictors. 

Statistical  analysis  of  the  survey  data  providtsf  initial  approximation.-, 
of  the  quantitative  aspects  of  the  software  factors  used  in  the  prediction 
methodology.  As  was  indicated  earlier,  the  participants  in  the  survey  were 
highly  qualified  professionals.  In  the  absence  of  precise  and  extensive 
measured  data,  their  collective  opinions  were  used  as  quantitative  estimates 
of  the  effects  of  software  factors  on  its  reliability. 

The  results  of  the  full-scale  survey  were,  therefore,  used  as  the 
basis  for  the  calculation  of  inherent  reliability  and  tlie  developmental 
factors  used  within  the  methodology.  For  purposes  of  using  the  met  fiodo  1  ogy , 
best,  worst,  and  nominal  values  are  presented  for  each  lacLor.  These 
values  were  statistically  derived  from  the  survey  results. 


5.0  RKLIABILITY  PREDICTION  METHODOLOGY 


5.1  Overview  and  Objectives 

System  reliability  prediction  is  essential  to  planners  and  designers 
of  military  systems.  Unfortunately,  current  methods  of  evaluation  eithc'r 
ignore  or  simpl i st ica 1 ly  incorporate  the  software  contribution  to  system 
reliability.  Those  methods  that  do  account  for  software  use  test  results 
as  a  means  of  mathematically  predicting  its  reliability.  In  the  event  that 
the  software  contribution  causes  system  reliability  to  degrade  below  accept¬ 
able  levels,  testing  must  be  continued  to  effect  improvements. 

The  objective  of  this  reliability  prediction  methodology  is  to  provide 
a  method  tor  predicting  system  reliability,  including  both  hardware  and 
software  effects  in  the  beginning  and  throughout  development.  Proper  use 
of  the  methodology  will  enable  both  the  procuring  agency  and  the  developer 
to  predict  operational  reliability  at  a  Lime  when  it  is  still  possible  to 
a  f  feet  It  . 

5.2  Approach 

Hardware  reliability  prediction  is  adequately  described  by  MIL-HDBK- 
217D  and  its  supporting  documentation;  therefore,  the  methodology  described 
here  concentrates  on  the  prediction  of  the  system's  software  component. 
Essentially,  the  prediction  of  software  module  reliabilities  is  based  on 
inherent  factors  of  their  operational  mission  and  size.  They  are  adjusted 
by  developmental  factors  estimated  from  the  design,  development,  and  test¬ 
ing  methods  that  will  be  employed  during  the  development  process.  These 
component  reliabilities  are  then  mathematically  combined  via  a  Markov  tech¬ 
nique,  which  accounts  for  the  duty  cycling  effects  of  logic  paths  that  are 
being  randomly  exercised  in  accordance  with  functional  mission  requirements. 
After  a  single  software  reliability  figure  is  obtained,  the  combined  reli¬ 
ability  is  computed  as  though  hardware  and  software  are  series  components 
of  the  overall  system. 

5.3  Re  su 1 1  s 

Performing  a  reliability  analysis  of  a  system  prior  to  its  development 
generally  involves  successive  system  decompositions.  This  continues  until 
a  component  level  is  achieved,  where  reliability  information  is  available, 
followed  by  matfiemat ica 1  reconstruction  of  the  system  to  determine  overall 
system  reliability.  Although  software  must  be  decomposed  and  evaluated 
differently  from  hardware,  the  methodology  is  similar.  System  decomposition 
IS  performed  in  accordance  with  currently  used  techniques.  When  the  de¬ 
composition  reaches  a  subsystem  that  requires  embedded  software,  the  soft¬ 
ware  must  be  identified  as  an  entity  and  treated  as  a  subsystem  component. 
While  the  rest  of  the  hardware  components  are  being  evaluated  via  classical 
methods,  the  software  portion  can  be  evaluated  using  the  methodology 
described  here.  Resulting  information  can  then  be  used  in  the  mathematical 
reconstruction  of  the  system  reliability  prediction.  This  approach 
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allows  the  independent  calculation  of  hardware  and  software  subsystem  reli¬ 
abilities,  with  a  minimal  divergence  from  classical  reliability  prediction 
techniques.  It  also  isolates  the  methodology  used  for  software  reliability 
prediction  so  that  it  can  be  easily  modified,  expanded,  or  replaced  as  new 
methods  become  available. 

The  remainder  of  this  section  describes  those  procedures  that  must  be 
followed  to  predict  hardware  and  software  system  reliability.  The  well- 
known  and  practiced  methods  of  MIL-STD-785B  are  implemented  for  the  entire 
system.  However,  software  is  considered  a  separate  entity  for  which  reli¬ 
ability  must  be  specified,  allocated,  predicted  and  assessed.  This  is 
illustrated  in  Figure  2. 
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PREDICTION,  MEASUREMENT,  AND  ASSESSMENT 


Figure  2.  Classic  Approach  -  New  Ingredient 


.3.1  System  Reliability 

System  reliability  is  the  probability  that  the  system  will  perform  its 
ntended  function  for  the  prescribed  mission  and  time  period  in  the  sped 
ied  operating  environment.  At  the  highest  level  of  decomposition,  it  is 
ecessary  to  define  the  specific  mission  that  the  system  is  required  to 
erform.  As  a  minimum,  most  systems  can  be  considered  to  be  either  iti  an 
perating  mode  or  a  nonoperating  mode.  Many  systems  operate  in  a  multimis- 
ion  or  multiphased  scenario.  Since  the  ability  of  a  system  to  perform  its 
ntended  function  is  conditional,  depending  upon  its  availability  at  the 
ime  when  the  function  is  required,  nonoperating  failure  mechanisms  cannot 


be  Ignored.  Multiphased  mission  scenarios  involve  the  requirement  to  per¬ 
form  functionally  distinct  operations  in  accordance  with  some  recurring 
sequence  and/or  frequency.  Conversely,  multimission  systems  may  success¬ 
fully  perform  a  given  mission  despite  a  failed  state  in  a  logically  parallel 
mission  mode.  This  latter  example  redefines  the  system  and  is  not  con¬ 
sidered  in  this  analysis. 

Figure  3a  depicts  a  single  mission  system  in  both  operating  and  non¬ 
operating  modes.  The  ratio  of  time  spent  in  each  mode  is  a  critical  para¬ 
meter  of  the  overall  system  reliability  analysis.  For  example,  if  the 
system  described  were  an  emergency  power  backup  system,  its  nonoperating 
time  would  far  exceed  its  operating  time.  Conversely,  if  it  were  a  sur¬ 
veillance  radar,  its  nonoperating  time  would  probably  be  limited  to  the 
minimum  required  for  periodic  maintenance  or  refurbishment. 


Nonoperating 


MiMlon  Operational 


Figure  3a.  Single  Mission  Systi 


Figure  3b  depicts  a  multiphased  system  that  is  either  nonoperating  or 
performing  one  of  its  required  missions.  Since  all  of  the  missions  are 
required  tor  overall  system  success,  they  must  be  logically  serialized  in 
reliability  calculations. 
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In  both  examples,  it  is  necessary  to  determine  individual  mode  reli¬ 
abilities  and  prorate  their  contribution  to  system  reliability  in  accordance 
with  the  expected  time  to  be  allocated  to  each  mode.  Specifically: 


i-0  for  nonoperating  mode 

i=l  for  mission  1 
o 
o 
o 

i=n  for  mission  n. 


5.3.2  Mission  Reliability 

From  the  previous  discussion,  it  is  clear  that  determining  overall 
system  reliability  is  critically  dependent  on  a  proper  evaluation  of  mission 
reliability  for  each  intended  mission  of  the  system.  In  fact,  reliability 
requirements  for  many  systems  are  specified  in  terras  of  specific  mission 
reliabilities.  This  is  particularly  evident  in  military  systems  that  have 
drastically  different  peacetime  and  wartime  missions.  Although  the  defini¬ 
tion  of  system  reliability  does  not  change  with  the  mission  type,  it  is 
quite  likely  that  the  motivation  for  requiring  reliability  information 
does.  Reliability  information  is  required  for  tactical  and  strategic  plan¬ 
ning,  and  mission  success  probability  is  the  primary  measurement  criterion. 
Reliability  information  is  also  required  for  logistics  and  maintenance 
planning.  However,  availability  becomes  the  primary  goal.  In  any  case, 
mission  reliability  must  be  determined,  whether  it  is  to  be  a  stand-alone 
prediction  or  an  intermediate  calculation  needed  to  determine  overall 
system  reliability  or  operational  readiness. 

The  need  for  a  clear  definition  of  specific  mission  functional  require¬ 
ments  cannot  be  overemphasized.  In  the  case  of  hardware  reliability  pre¬ 
dictions,  distinct  missions  could  very  likely  have  considerably  different 
stress  and  environmental  influences  acting  upon  the  components,  thereby 
drastically  affecting  their  individual  reliabilities.  Furthermore,  the 
tuncLional  requirements  of  a  particular  mission  might  require  the  inclusion 
of  components  or  subsystems  not  required  in  another  mission.  Based  on  the 
physical  construction  of  the  system,  certain  components  might,  therefore, 
be  in  a  nonoperating  state  while  others  are  operational.  The  resulting 
reliability  block  diagram  could  be  considerably  altered. 
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Hardware  duty  cycliog  effects  are  generally  considered  in  reliability 
predictions  at  the  highest  levels  only.  Individual  components  within  a 
circuit  are  usually  considered  to  be  all  operational  or  all  dormant.  Since 
each  component  is  essential  to  the  continuity  of  the  circuit,  duty  cycling 
effects  are  considered  negligible  and  consequently  ignored. 


Duty  cycling  effects,  however,  are  probably  the  most  critical  deter¬ 
minant  of  software  reliability.  Software  cannot  fail  unless  it  is  being 
used.  A  logic  path  does  not  exist  during  a  time  period  (albeit  micro¬ 
seconds)  when  it  is  not  being  used.  The  functional  flow  of  control  through 
a  computer  program  is,  in  fact,  its  reliability  block  diagram.  This  repre¬ 
sents  the  first  and  most  critical  deviation  from  classical  reliability 
techniques,  which  is  necessitated  by  the  inclusion  of  software  within  a 
system  and  software  reliability  calculations  within  a  reliabilty  analysis. 
It  follows  that  software  reliability  is  a  function  of  individual  component 
reliabilities  and  their  path  probabilities.  The  method  for  determining 
each  will  be  described  later  in  this  report. 


Mission  decomposition  is  performed  as  described  by  classical  relia¬ 
bility  modeling  techniques; 


_l^  Define  the  subsystems  required  for  mission  success,  and  translate 
this  into  a  mission  success  diagram. 


For  each  subsystem,  continue  the  decomposi t ion  process  until  a 
level  is  reached  where  a  reliability  prediction  can  be  made. 


After  subsystem  reliability  predictions  (including  software)  have  been 
made,  determination  of  mission  reliability  is  the  mathematical  reverse  of 
Che  decomposition  procedure.  Figure  4  depicts  a  typical  mission  reliability 
prediction  analysis.  It  shows  a  missile  system  comprised  of  six  major 
components,  two  of  which  contain  embedded  software.  Although  Che  figure 
only  shows  two  levels  of  decomposition,  it  is  obvious  that  Che  process  can 
be  continued  as  far  as  necessary  Co  arrive  at  a  level  where  subsystem  or 
component  reliability  can  be  predicted. 


Two  things  should  be  emphasized  at  this  point.  First,  the  presence  of 
software  in  the  reliability  block  diagram  does  not  alter  Che  normal  tech¬ 
niques  used  for  system  decomposition,  nor  does  it  affect  the  mathematical 
reconstruction  of  Che  system  reliability  prediction.  Only  a  technique  for 
decomposing  component  parts  and  evaluating  their  individual  and  collective 
reliabilities  is  needed. 


The  second  point  is  a  major  one.  As  shown  later,  the  software  reli¬ 
ability  prediction  methodology  relies  very  heavily  on  the  functional  cycl¬ 
ing  of  the  individual  software  modules.  It  is,  therefore,  strongly  mission 
dependent.  To  combine  software  and  hardware  reliability  ca Iculat ions,  it 
is  essential  that  the  units  of  measurement  be  consistent.  As  shown  later, 
the  software  model  will  produce  reliability  calculations  for  a  specific 
mission  and  mission  time.  Unlike  hardware  considerations,  which  assume 
that  the  components  have  arrived  at  a  level  of  constant  failure  rate,  the 
software  failure  rate  can  be  considered  constant  only  for  a  prescribed 
mission  scenario.  Separate  analysis  must  be  performed  for  each  mission 
considered  in  the  analysis. 
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where: 


Missile  Reltabimy 

R  =  Ra  X  (  Rb  -f  Rc  -  RbRc  )  x  Rd  x  Re  X  R1 

Ra  =  Ra1  X  Ra2  x  Ra3 

Rb  =  Rbl  X  Rb2 

Rc  =  Rcl 

Rd  =  Rdl  X  Rd2  x  Rd3 

Re  =  Rel  x  Re1  x  Rel  x  Ret  x  Re2  x  Re2  x  Re2  x  Re2 
Rf  =  Rfl  X  Rf2 


Figure  4.  Typical  Mission  Reliability  Fred  i  c  L  i  on  Aiipru.tcli 


5.3.3  Hardware  Reliability 

Hardware  reliability  prediction  techniques  an'  well  e  ,st  .ili  I  i  .shed  .jud 
generally  well  understood  and  practiced.  The  electronic  r.iling.s  .inil  leli 
ability  evaluation  miithods  prestjnted  in  M I  L~H))BK-2 1  71)  are  a.ssuimul  lo  he 
completely  applicable  to  the  hardware-  portion  ol  the  combine. I  h.irdw.ite/ 
solLware  methodology.  Generally,  hardware  components  have  Immoi  c.ii  .•goi  i /e.l 
into  de-vice  types,  each  of  which  has  a  descriptive  model  loi  de  i  e  i  m  i  n.i  l  i  on 
ot  lib  rel  labi  1  ity  . 
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Each  specific  component  in  a  device  category  has  a  base  failure  rate 
that  represents  its  design  characteristics.  Multipliers,  or  pi  factors  are 
applied  to  the  base  rate  as  specified  in  the  device  model  to  arrive  at  a 
component  failure  rate  adjusted  by  its  developmental  and  operational  char¬ 
acteristics  (Figure  5). 


ELECTRONIC  SUBSYSTEM 
I  COMPONENT  A  (ICI - 


■  COMPONENT  B  (DIODE)  - 


-  COMPONENT  C  (RESISTOR  )- 


COMPONENT  D  (CAPACITOR)  - 


-  f (bat*  raw.  Pi  factors) 
Xq  <■  flbata  rata.  Pi  factors) 
X^  ~  f(bass  rats.  Pi  factors) 
Xq  -  fibasa  raw.  Pi  factors) 


TOTAL  FAILURE  RATE  -  X^  *  ^C  ^D 


Reliability  °  e 


-  (Total  Failure  Rate)  x  (time) 


Figure  5.  Hardware  Reliabilit; 


5.3.4  Software  Reliability 

The  methodology  developed  for  predicting  software  reliability  is 
organized  to  maximize  the  procedural  similarities  to  hardware  techniques. 

In  hardware  calculations,  it  is  essential  to  identify  the  component  parts, 
determine  their  respective  reliabilities  using  base  failures  rates  and 
system/application  specific  multipliers,  and  mathematically  combine  them  in 
accordance  with  the  reliability  block  diagram.  The  software  methodology 
uses  a  parallel  sequence  of  events:  1)  the  software  subsystem  is  decomposed 
into  its  components,  2)  for  each  component,  a  reliability  prediction  is 
calculated  based  on  its  inherent  characteristics  and  developmental  factors, 
and  3)  the  component  reliabilities  are  mathematically  recombined  into  a 
single  prediction  for  the  overall  subsystem.  Figure  6  summarizes  the  over¬ 
all  flow  of  the  methodology.  Using  standard  software  engineering  tools, 
the  following  sequence  of  events  is  accomplished. 


Functional 

DacompoaKlon 

Procassing 

Characteristics 

Context  Flow 
Analyele 


Contponent  A 

Component  B 

e 


Inherent  Reliability  Characterletics  of  A 
Inherent  Reliability  Characteristics  of  B 


Component  n 


Inherent  Reliability  Characteristics  of  n 


Development 

Characteristics 


Expected  Reliability  of  Component  A 
Expected  Reliability  of  Component  B 


Expected  Reliability  of  Component  n 


Functional 
Thread  Analysis 


Software  Subsystem  Reliability 
(Mission  Cycle  Based) 


Mission  Timing 
Analysis 


Software  Subsystem  Reliability 
(Mission  Time  Based) 


Figure  6,  Software  Reliability  Prediction  Methodology 

1  A  functional  decomposition  is  performed  to  identify  those  physical 
components  (programs,  modules,  routines,  etc.)  that  comprise  the 
software  subsystem. 

2  Analysis  of  the  processing  characteristics  of  each  component  and  an 
analysis  of  the  overall  context  (control  and  data  information)  flow 
between  components  is  performed  to  identify  their  inherent  charac¬ 
teristics.  Each  component  is  classified  in  accordance  with  its 
ciiaracter ist  ICS  (e.g.,  real-time,  single-function,  extensive  inter¬ 
face  s ,  etc.). 

3  Characteristics  of  the  intended  development  process  are  identified 
and  categorized  by  the  nature  of  their  contributions  to  the 
process.  As  discussed  earlier,  all  methods  and  techniques  are 
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categorized  as  being  either  error  avoidance  mechanisms  (e.g.,  use 
of  a  high  order  language,  use  of  structured  techniques,  etc.)  or 
error  detection  mechanisms  (e.g.,  frequent  walkthroughs,  quality 
audits ,  etc.). 

^  Individual  software  component  reliabilities  are  computed. 

^  A  mission  functional  thread  analysis  is  performed  to  determine  the 
duty  cycling  effects  caused  by  the  specific  mission  profile  for 
which  reliability  prediction  is  needed. 

^  Overall  software  reliability  for  the  specific  mission  is  computed 
using  a  Markov  technique. 

T_  Overall  system  reliability  for  the  specific  mission  is  computed  as 
the  product  of  hardware  and  software  reliabilities. 

All  of  the  above  data  can  be  made  available  during  the  early  stages  of 
a  software  development.  It  is  possible  to  perform  all  of  the  above  analy¬ 
ses  based  on  proposal  or  preliminary  design  information.  Therefore,  it  is 
also  possible  to  predict  software  reliability  at  a  time  when  it  is  still 
possible  to  alter  development  plans  to  enhance  it.  The  early  application 
of  the  methodology  also  provides  a  basis  for  comparing  the  cost  effective¬ 
ness  of  alternate  approaches  to  increasing  reliability.  Of  course,  as  the 
project  progresses,  more  detailed  design  information  and  more  precise 
timing  estimates  become  available  to  the  analyst.  Periodic  reapplication 
of  the  methodology  can  be  used  to  give  assurance  that  reliability  require¬ 
ments  will  be  met  or  to  alert  management  to  avoid  potential  shortfalls. 
Actual  measurements  of  component  reliability  can  also  be  made  during 
development,  and  the  system  reliability  can  be  estimated  prior  to  software 
integrat ion . 

The  remainder  of  this  section  discusses  the  specific  implementation  of 
the  methodology.  This  volume  presents  the  rationale  and  derivation  of  the 
model.  Volume  II  includes  detailed  examples  as  well  as  the  list  of  factors 
derived  from  the  surveys  conducted. 


Functional  Decomposition 

Decomposition  of  computer  software  is  accomplished  much  like  hardware. 
Whereas,  hardware  subsystems  can  be  segmented  wherever  a  connection  has 
been  or  will  be  made,  software  can  be  broken  anywhere  in  the  sequence  of 
commands  that  it  executes.  In  both  cases,  however,  it  is  illogical  to 
disconnect  components,  except  at  the  physical  (or  logical)  boundaries  of 
complete  subunits.  System  hardware  might  be  decomposed  into  black  boxes', 
the  boxes  decomposed  into  printed  circuit  boards;  the  boards  decomposed 
into  circuits;  and  finally,  the  circuits  decomposed  into  their  respective 
electrical  components.  It  is  essential  that  every  phase  of  the  process 
yields  complete  subunits.  Computer  programs  are  similarly  decomposed. 


As  oae  of  Lhe  largest  consumers  of  computer  software,  the  Department 
of  Defense  has  taken  a  leadership  role  in  the  development  of  software 
standards  and  methods.  Figure  7  illustrates  the  generally  accepted  terrain 
ology  associated  with  software  decomposition.  It  lists  the  terminology 

COMPUTER  SOFTWARE  CONFIGURATION  ITEM  (CSCI)  -  software  aggregate  which  is 
designated  by  the  procuring  agency  for  configuration  control 

1 

I 

' - COMPUTER  SOFTWARE  COMPONENT  (CSC)  -  a  functional  or  logically 

distinct  part  of  a  computer  software  configuration  item 

I 

I 

I 

* - UNIT  -  the  lowest  level  logical  entity  specified  in  the 

detailed  design  which  completely  describes  a  non- 
divisible  function  in  sufficient  detail  to  allow 
implementing  code  to  be  produced  and  tested  independently 
of  other  units 


^--MODULE  -  the  lowest  physical  entity  specified  in  the 
detailed  design  which  may  be  assembled  or  compiled 
alone 

I 

i 

^ — INSTRUCTION  -  a  single  line  of  code  which  may 
correspond  to  a  single  action  of  the  computer 
or  may  be  automatically  translated  into  a 
series  of  single  actions  of  the  computer 


|-- OPERATION  -  the  action  to  be  performed  by 
j  the  computer 


■"OPERANDS  -  the  symbolic  or  absolute 

addresses  of  Lhe  computer  memory  where  Lhe 
data  to  be  processed  reside. 


Figure  7.  Software  Decomposition  Process  and  Terminolo^ 
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specified  ia  Che  proposed  military  standard  DoD-STD-SDS  and  has  been 
extended  to  Che  lowest  possible  level.  At  the  highest  level,  software  is 
defined  as  a  configuration  item,  one  of  which  is  defined  by  and  for  Che 
procuring  agency.  It  has  considerable  contractual  significance,  but  no 
logical  or  functional  characteristics.  At  the  other  end  of  Che  spectrum, 
the  level  of  detail  is  so  specific  that  prediction  is  not  possible  until 
after  implementation. 

Although  Che  software  prediction  methodology  is  not  affected  by  Che 
level  to  which  Che  software  is  decomposed,  it  is  practical  to  define  its 
applicability  as  ranging  from  Che  CSC  level  through  the  module  level. 
Generally,  CSC  level  decomposition  is  possible  during  the  requirements 
definition  phase  of  software  development;  unit  decomposition  is  possible 
during  preliminary  design;  and  module  decomposition  is  possible  during  the 
detailed  design  phase.  At  each  milestone,  the  software  reliability  predic¬ 
tion  methodology  can  be  reapplied  with  greater  accuracy.  For  generality, 
the  term  "software  component"  is  used  to  include  all  levels  of  software 
decomposition  between  the  CSC  and  module  level. 

Inherent  Reliability  Characteristics 

Just  as  hardware  components  can  be  classified  into  component  categories 
such  as  resistors,  capacitors,  and  diodes,  software  can  be  categorized  into 
characteristic  groups.  In  the  case  of  software,  however,  the  dLstinctton 
between  groups  is  based  on  logical  composition  rather  than  physical  makeup. 

A  direct  relationship  between  the  complexity  of  a  computer  program  and  its 
reliability  is  intuitively  expected.  Likewise,  it  is  intuitive  that  the 
complexity  of  the  software  is  related  to  its  intended  application  (the  pro¬ 
grammatic  complexity  of  the  design  is  considered  later).  In  other  words, 
even  before  it  is  designed  or  implemented,  real  time  software  is  expected 
to  be  more  error  prone  than  batch  software  where  timing  is  not  a  critical 
consideration.  Similarly,  historical  evidence  shows  that  programs  with  a 
large  amount  of  interface  requirements  experience  higher  failure  rates  than 
those  that  contain  minimal  interfaces. 

To  determine  the  inherent  reliability  characteristic  of  a  given  module, 
the  analyst  must  identify  those  operational  characteristics  that  best 
describe  the  module  to  b^  developed.  Error  distribution  estimates  for  the 
more  predominant  operational  characteristics  of  software  modules  have  been 
derived  from  the  surveys  described  earlier.  At  present,  they  accurately 
represent  the  combined  opinions  of  software  experts.  Hopefully,  the 
recently  increased  emphasis  on  software  reliability  and  quality  measurement 
will  promote  and  facilitate  the  collection  of  detailed  statistical  information 
that  IS  similar  to  that  information  for  the  hardware  reliability  engineer 
in  MIL-HDBK-21 7D.  Currently,  the  methodology  is  more  urgently  needed  than 
the  data  precision. 


It  should  be  noted  that  this  phase  of  analysis  addresses  the  inherent 
error  characteristics  of  the  various  components  which  must  be  combined  with 
the  planned  development  process  characteristics  to  determine  module  reli¬ 
ability.  The  calculation  of  the  module  characteristic  error  distribution 
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is  simply  an  average  of  Che  effects  of  its  inherent  characteristics.  If  we 
define  C(j)  as  Che  percentage  of  errors  of  type  j  to  be  expected  in  Che 
module  being  evaluated,  it  follows  that: 


i=N 

c<i.  -  2 

i  =  l 


c(j,i) 

N 


(2) 


where:  c(j,i)  is  the  percentage  of  errors  of  type  j  caused 

by  inherent  characteristic  i,  (listed  in  Volume  II) 

and  N  is  the  number  of  inherent  characteristics  applicable. 

Development  Characteristics 

As  described  earlier,  software  development  characteristics  can  be 
categorized  in  terms  of  their  contributions  to  error  avoidance  or  error 
detection.  Virtually  any  activity  during  the  development  life  cycle  can  be 
evaluated  in  terms  of  these  two  characteristics.  The  software  reliability 
prediction  methodology  uses  measures  of  error  avoidance  and  error  detection 
effectiveness  which  are  based  on  the  planned  technical  and  managerial 
techniques  and  methods  used  to  develop  the  software.  As  in  the  case  of 
inherent  reliability  characteristics,  the  data  base  used  for  these  predic¬ 
tions  was  constructed  from  the  results  of  the  survey  performed  during  the 
study. 

It  is  generally  accepted  that  certain  development  techniques  are  good 
and  will  make  the  software  better.  For  example,  it  is  generally  agreed 
that  structured  approaches  are  good  and  will  have  a  positive  influence  on 
the  quality  and  reliability  of  the  product.  Quantification  of  the  effects 
is  typically  attempted  after  the  development  is  complete,  and  the  results 
are  rarely  applicable  to  new  projects.  While  the  approach  described  herein 
has  a  limitation  due  to  the  unavailability  of  detailed  historical  records, 
the  method  is  directly  applicable  to  any  software  development  venture. 

trror  avoidance  effectiveness  is  calculated  as  the  probability  of  not 
introducing  an  error  given  the  opportunity  tor  making  the  error.  Mathe¬ 
matically,  It  is  computed  as  unity  minus  the  probability  that  a  hypothetical 
error  will  not  be  avoided  by  any  of  the  techniques  employed.  If  we  define 
A( j )  as  the  probability  of  avoiding  errors  of  type  j,  it  follows  that: 


i=N 

A(  j: 

)  =  1.00 

1 

o 

o 

1 

i)) 

(3) 

1=1 

where : 

a(j,i)  IS  the  probability  of 

error  type 

j  being  avoided  by 

the  application  of 

technique 

1 ,  (listed 

in  Volume  11) 

and 

N  IS  the  number  of 

techniques 

emp 1 oyed . 
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Error  detection  effectiveness  is  similarly  calculated  as  the  prob¬ 
ability  that  an  existing  error  will  be  discovered  and  corrected.  Again,  it 
is  mathematically  computed  as  unity  minus  the  probability  that  a  hypotheti¬ 
cal  error  will  not  be  detected  by  any  of  the  techniques  employed.  If  we 
define  D(j)  as  the  effectiveness  of  detecting  errors  of  type  j,  it  follows 
that : 


D(j)  =  1.00  - 


(1.00  -  d(j,i)) 


where:  d(j,i)  is  the  probability  of  error  type  j  being  detected 

by  the  application  of  technique  i,  (listed  in  Volume  II) 

and  N  is  the  number  of  techniques  employed. 

Expected  Component  Reliability 

The  expected  reliability  of  a  software  component  is  a  function  of  the 
inherent  characteristics  of  the  component  and  the  characteristics  of  the 
development  process  used  to  produce  it.  Unfortunately,  there  is  insufficient 
historical  data  available  to  isolate,  with  any  degree  of  confidence,  the 
casual  relationship  between  a  specific  characteristic  or  development  tech¬ 
nique  and  the  resulting  effect  on  component  reliability.  The  term  "reliabil¬ 
ity"  is  used  here  to  connotate  the  probability  that  the  software  component 
will  perform  its  intended  functions  correctly  the  next  time  it  is  executed. 
That  is,  component  reliability  is  defined  as  the  probability  of  success  in 
a  single  trial.  If  it  were  possible  to  extensively  exercise  the  component 
in  a  controlled  experiment,  its  reliability  could  be  approximated  by  the 
ratio  of  its  successful  executions  to  its  total  executions: 


Rj,  =  Pj.  (success)' 


Number  of  Successful  Executions 
Number  of  Executions 


Unfortunately,  the  component  does  not  exist  at  the  beginning  of  the  develop¬ 
ment  effort,  so  it  is  not  possible  to  determine  Che  success  probability  as 
a  simple  relative  frequency  of  measured  events.  The  logical  recourse  is  to 
theoretically  derive  the  probability  function  by  computing  Che  ratio  of  all 
possible  successful  executions  to  the  number  of  possible  executions.  It 
should  be  noted  that  the  number  of  possible  executions  is  extremely  large 
since  every  variation  of  Che  inputs  to  the  component  and  every  variation  of 
the  computer  memory  creates  a  new  combination  of  circumstances  in  which  Che 
component  must  operate. 

We  can  redefine  the  probability  of  success  in  terras  of  the  number  of 
variations  as: 


R  =  P  (success)  = 
c  r 


Number  of  Possible  Successful  Variations 
Total  Number  of  Possible  Variations 


In  the  above  expression,  the  number  of  variations  is  an  extremely  large 
number  representing  all  combinations  of  inputs,  memory  states  and  functional 
requirements  to  be  accomplished. 
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Although  the  component  itselt  does  not  yet  exit,  its  functional  requir<’ 
ments  do.  Likewise,  the  process  or  mechanism  exits  to  convert  those  require 
ments  into  a  software  component.  The  software  engineer  is  required  to 
create  logic  which  will  correctly  perform  every  possible  variation.  Those 
that  he  properly  implements  will  successfully  execute;  those  that  lie  fails 
to  implement  correctly  will  not.  The  ratio  can  be  expressed  as: 


R  = 
c 


( 


success 


, )  -  Number  of  Variations  Correctly  Implemented 


Total  Number  of  Variations  Implemented 


(7) 


This  is  obviously  a  measure  of  the  probability  of  successful  implementa¬ 
tion  well  as  the  probability  of  successful  execution. 


AT  SOMK  LLVEL  OF  DECOMPOSITION,  THE  RELIABILITY 
OF  THE  PRODUCT  IS  EQUAL  TO  THE  RELIABILITY  OF 
THE  PROCESS  WHICH  PRODUCED  IT. 


It  follows  that  developmental  techniques  which  increase  the  reliability 
of  the  process  will  also  increase  the  reliability  of  the  product  by  reduc¬ 
ing  its  error  content.  That  is,  component  reliability  can  be  predicted 
based  on  an  inherent  probability  of  successful  implementation  enhanced  by 
the  application  of  the  error  avoidance  and  detection  techniques  discussed 
in  the  Development  Characteristics  section.  Specifically,  software  com¬ 
ponent  reliability  can  be  calculated  as: 


R^  =  Rj^  +  E(l-Ri) 


(8) 


where:  R^  is  the  component  reliability  (probability  of 

succe  s  s ) 


Rj^  IS  the  inherent  reliability  of  either  the  process  or 
the  component 


and 


E  IS  the  enhancement  factor  achieved  by  the  application  of 
error  avoidance  and  detection  techniques. 


a.  Inherent  Component  Reliability  -  Rj^ 


There  are  several  approaches  to  determine  inherent  software  component 
reliability,  each  of  which  has  both  advantages  and  disadvantages.  Any 
measure  which  presents  a  ratio  of  successful  implementations  to  total 
implementations  may  be  used.  Some  of  the  more  obvious  measures  are  discus¬ 
sed  below: 


Assume  Chat  is  equal  Co  zero.  This  causes  equal  ion  (8)  Co 
reduce  simply  to  the  enhaacement  factor.  At  first  glance,  this 
approach  appears  Co  be  a  gross  simplification.  It  essentially 
says  that  unless  an  effort  is  made  to  avoid  and/or  detect  errors, 

Che  software  component  will  not  work.  We  feel  that  this  is  the 
most  theoretically  sound  approach.  However,  it  assumes  that  the 
developer  has  absolutely  no  knowledge  of  the  product  he  is  respon¬ 
sible  to  develop.  Even  a  casual  knowledge  of  what  he's  supposed  to 
do  can  be  considered  an  error  avoidance  technique.  This  assumption 
carries  with  it  the  additional  assumption  that  the  checklist  of 
avoidance  and  detection  techniques  is  exhaustive.  It  does,  however, 
define  a  lower  bound  on  the  reliability  prediction  when  the  devel¬ 
opmental  characteristics  are  known. 

2  Assume  that  the  inherent  UNreliability  is  proportional  to  measured 
fault  densities  of  existing  software  which  has  similar  characteris¬ 
tics.  This  approach  has  Che  advantage  of  data  availability.  Al¬ 
though  there  is  a  fairly  wide  range  of  measured  values  of  faults 
per  line  of  code,  there  is  sufficient  historical  data  for  an  analyst; 
to  make  a  sound  engineering  determination  of  the  best  figure  to 
use.  Care  must  be  taken,  however,  to  distinguish  how  and  when  the 
data  used  was  collected.  Many  organizations  do  not  begin  counting 
faults  until  the  software  is  tested  in  the  overall  system  while 
others  begin  recording  failure  data  as  soon  as  individual  components 
have  completed  unit  test.  The  prediction  methodology  assumes  that 
includes  consideration  of  all  errors  made,  not  just  the  ones 
recorded  subsequent  to  integration  testing.  This  method  should 
produce  an  upper  bound  on  Che  reliability  prediction  due  to  Che 
fact  Chat  the  actual  number  of  faults  in  a  software  product  cannot 
be  less  than  the  number  recorded. 

3^  Assume  that  Che  inherent  UNreliability  is  proportional  to  a  fault 
density  which  has  been  interpolated  from  the  range  of  historically 
recorded  fault  densities.  The  interpolation  could  be  based  on  the 
same  characteristics  already  discussed  in  the  Inherent  Reliability 
Character  i  St  ict.  section.  Although  such  a  scheme  has  not  yet  been 
formulated,  it  is  the  opinion  of  the  author  that  one  could  be 
created  and  Chat  it  would  provide  the  most  unbiased  measure. 
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b .  Reliability  Enhancement  Factor  -  E 

Figure  8  illustrates  the  relationship  between  inherent  reliability, 
avoidance  effectiveness  and  detection  effectiveness.  The  figure  introduces 
some  terminology  not  previously  described; 
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Figure  8 .  Relationship  of  R(I),  R(C),  A  and  D 


Inherent  reliability. 

Total  possible  variations  implemented  (reference  equation  (6)) 

Total  variations  inherently  implemented  correctly.  These  are 
the  variations  that  would  have  been  properly  implemented 
without  process  enhancement. 

Total  variations  inherently  implemented  incorrectly.  These 
are  candidates  to  be  avoided  or  detected. 

Number  of  variations  being  worked  on  during  the  i'th  iteration 
These  include  the  original  errors  to  be  eliminated  plus 
reworks  of  errors  discovered  on  the  previous  iteration. 

Number  of  variations  which  "pass  thru"  the  avo idance/de t ec t lon 
filters  because  they  are  already  correctly  implemented. 

Number  of  previously  incorrect  implementations  which  were 
successfully  avoided  on  the  current  iteration. 


Number  of  previously  incorrect  implementations  which  were 
neither  avoided  nor  detected  on  the  current  iteration. 

Number  of  previously  incorrect  implementations  which  were 
successfully  detected  and  returned  for  rework. 

These  are  the  error  avoidance  and  detection  factors  described 
in  the  Development  Characteristics  section. 


The  process  depicted  represents  a  typical  software  development  opera¬ 
tion.  As  a  result  of  the  inherent  characteristics  of  the  software  to  be 
developed,  errors  will  be  made.  The  development  team  will  attempt  to  avoid 
making  those  errors  by  the  application  of  software  engineering  techniques. 
Recognizing  that  they  will  probably  not  avoid  all  errors,  tests  and  other 
detection  techniques  are  implemented  to  locate  and  rework  the  faults. 
Avoided  errors  will  exit  the  process  as  corrected  implemented  variations. 
Detected  errors  will  be  reworked  by  the  process  until  they  either  are 
avoided  or  escape  the  detection  mechanisms.  Eventually,  all  N  variations 
exit  the  process.  Since  the  enhancement  factor  is  an  improvement  factor, 
it  is  defined  as: 


g  _  NBG _  _  Number  of  Corrected  Bad  Implementations 

NBG  +  NBB  Total  Number  of  Bad  Implementations 

The  derivation  that  follows,  verifies  that  the  enhancement  factor  is 
independent  of  the  initial  number  of  bad  implementations  and  is  simply  a 
function  of  the  process  characteristics: 


E 


A 

1  -  D(l-A) 


(10) 


where:  A  and  D  are  the  error  avoidance  and  detection  factors 

described  in  Section  5.3.4. 


(1)  Preliminary  Calculations 

The  following  relationships  can  be  determined  directly  from  Figure  8. 
The  number  of  initial  inputs  to  the  process  is  equal  to  the  initial  number 
of  errors  expected  due  to  inherent  characteristics.  On  subsequent  itera¬ 
tions,  the  number  of  inputs  is  equal  to  the  number  of  reworks  necessitated 
by  the  previous  iteration: 

Iq  =  NB  and  Ij^  =  NBD£_p  (11) 

The  number  of  avoided  errors  on  any  iteration  is  equal  to  the  number 
of  inputs  processed  multiplied  by  the  probability  of  avoiding  errors: 

NBGi  =  Ii(A).  (12) 

The  number  of  errors  detected  on  any  iteration  is  equal  to  the  number 
of  errors  NOT  avoided  multiplied  by  the  probability  of  detecting  them: 


NBDi  =  Ii(l-A)(D).  (13) 

The  number  of  errors  not  detected  on  any  iteration  is  equal  to  the 
number  of  errors  NOT  avoided  multiplied  by  the  probability  of  NOT  detecting 
them: 

NBB^  =  l^(l-A)(i-D).  (14) 
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Combining  equations  (11)  and  (13)  yields  the  feedback  relationship 
which  causes  the  process  to  continue  until  all  of  the  original  inputs  have 
exited  the  process: 


I.  =  Iq[D(1-A)]\ 


(15) 


(2)  Derivation 


The  components  of  equation  (9)  can  now  be  expressed  in  terms  of  the 
initial  inputs  to  the  process  and  the  avoidance/detection  mechanisms  used 
during  the  process. 


The  number  of  corrected  at.d  uncorrected  bad  implementations  which  exit 
the  process  are  equal  to  the  infinite  sum  of  the  number  which  exit  on 
individual  iterations.  That  is: 


NBBi  . 


NBG  =  NBG^  and  NBB  = 

i=0  i=0 

Substitution  of  equations  (13)  and  (14)  into  (15)  yields: 
00  00 


(16) 


NBG  = 


Ii(A)  and  NBB  =  I^d- 


A)(l-D). 


(17) 


i=0 


i=0 


Since  the  multipliers  are  independent  of  i,  they  can  be  factored  out 
of  the  summations: 


00 


NBG  =  (A)  >'  and  NBB  =  (1-A)(1-D) 


00 

2  ‘i- 


i=0 


i=0 


It  follows  that  equation  (9)  becomes: 


E  =  — 


(A)  2,  4 

i=0 


00 


00 
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(A)  1^  +  (1-A)(1-D) 
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The  summaiLons  cancel  reaulting  in: 


A  +  (i-A)(l-D) 


120) 


o  r : 


1  -  D(l-A)  ■ 


(21) 


Path  Analysis 

After  the  reliability  of  each  software  component  has  been  estimated, 
it  is  possible  to  predict  overall  software  subsystem  reliability.  This  is 
accomplished  by  using  the  Markov  process  as  suggested  by  Cheung  [36].  The 
approach  is  based  on  the  fact  that  individual  software  components  contri¬ 
bute  to  the  overall  reliability  when,  and  only  when,  they  are  execut.jd.  It 
has  already  been  shown  that  individual  component  reliabilities  can  be  pre¬ 
dicted.  Fortunately,  it  is  also  possible  to  predict  their  operational 
usage.  As  was  mentioned  earlier,  preliminary  design  activities  not  only 
identify  the  required  software  functional  components,  but  also  show  their 
relationships  to  each  other.  This  may  be  expressed  in  the  form  of  function¬ 
al  flow  diagrams,  decision  tables,  hierarchical  structure  charts,  or  in  the 
case  of  interrupt  driven  system,  timing  and  frequency  requirements  for  each 
component . 

The  flow  of  control  between  software  program  components  can  be  con¬ 
sidered  a  Markov  process  if  we  assume  that  component  reliabilities  are 
independent.  If  a  given  software  program  has  n  components,  it  is  necessary 
to  know  the  reliability  of  each  component  and  the  probability  of  going  from 
one  component  to  another.  The  component  reliabilities  are  in  the  diagonal 
matrix  R,  and  the  path  probabilities  are  in  the  matrix  P;  i.e., 
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The  matrix  Q  is  the  product  of  matrices  R  and  P.  The  ij  th  entry  rep 
resents  the  joint  probability  that  component  L  will  execute  correctly  (R^) 
AND  pass  control  to  component  j  (P^j). 
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By  considering  each  component  to  be  a  state  and  by  defining  two 
additional  states,  C  for  correct  program  termination  and  F  for  failed 
termination  of  the  program,  a  Markov  chain  can  be  constructed  with  n+2 
states.  The  transition  matrix  T  is  formed  by  adding  two  rows  and  two 
columns  to  the  matrix  Q.  An  additional  two  rows  and  columns  are  for  the 
states  C  and  F.  The  matrix  T  is  defined  as  follows; 
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The  ij'th  entry  of  T  is  the  probability  of  going  from  state  i  to  stat 
j  in  one  step.  The  ij'th  entry  of  T*T  is  the  probability  of  going  from 
state  i  to  state  j  in  two  steps.  The  ij'th  entry  of  T^T'^T  is  the  probabil 
ity  of  going  from  i  to  j  in  three  steps.  The  reliability  of  the  software 
IS  the  probability  of  going  from  state  1  to  state  C  in  x  steps  or  less,  as 
X  approaches  infinity.  Thus,  to  compute  this  reliability,  calculate 


as  X  approaches  infinity. 


There  is  a  simpler  way  to  compute  the  reliability  of  a  program  using 
the  matrix  Q.  Let 

S  =  I  +  Q  +  -t-  .  .  .  (2  5) 

where  I  is  the  identity  matrix.  Note  that 

(I-Q)  *  (I  +  Q  +  Q^+  Q^+  +  .  .  .)  =  I  +  Q  +  Q^+  Q^.  .  . 

-  Q  -  .  .  .  (26) 


and  so, 


2  3  4 

I+Q  +  Q  +  Q  +  Q  +  . 


(I  -  Q) 


It  follows  that 


S  =  (1  -  Q)  .  (2b) 

Letting  be  the  entry  in  the  first  row  and  n'th  column  of  S,  the 

reliability  of  the  program  for  a  single  cycle,  R^, ,  is  given  by 

c  In  n  (29) 

Several  important  points  should  be  made  here.  First,  since  the  calculation 
of  software  reliability  was  computed  on  operational  path  probabilities,  the 
prediction  made  is  applicable  only  for  the  scenario  or  specific  mission 
described  by  these  probabilities.  In  the  likely  event  that  the  system 
being  analyzed  has  a  variety  of  missions  (e.g.,  peacetime,  standby,  war¬ 
time,  etc.),  the  reliability  of  each  must  be  independently  calculated  by 
adjusting  the  path  probabilities  matrix  and  reaccomplishing  the  Markov 
analysis.  A  second  point  is  equally  significant,  but  tends  to  simplify  the 
overall  prediction.  The  numerical  value  of  the  software  reliability,  is 
independent  of  time.  It  was  computed  as  the  function  of  a  specific  mission 
or  operational  scenario  and  can  be  assumed  to  be  constant  for  that  mission. 
The  final  step  in  predicting  a  combined  hardware/soft  ware  system  reliability 
value  is  calculated  by  multiplying  the  hardware  reliability  value,  deter¬ 
mined  by  classical  methods  and  the  software  reliability  value,  calculated 
by  the  methods  described  herein. 

5.4  Conclusions 

The  methodology  described  in  this  report  was  developed  to  allow  the 
independent  calculation  of  subsystem  reliabilities  for  hardware  and  soft¬ 
ware  components  of  an  overall  system  and  the  combination  of  these  calcula¬ 
tions  into  a  single  reliability  prediction.  For  the  hardware  subsystem, 
the  classical  methods  of  MIL-HDBK-2 1 7D  are  used  without  modification.  For 
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Che  software  subsystem,  a  new  approach  is  developed,  which  combines  soft¬ 
ware  knowledge  with  influences  caused  by  the  development  methodologies 
employed.  Knowledge,  or  inherent  reliability,  is  based  on  historical  reli¬ 
ability  measurements  of  software  developed  for  similar  applications.  The 
influence  or  pi  factors  are  determined  by  the  manner  in  which  the  software 
will  be  or  is  being  developed. 

This  approach  allows  the  system  planners  to  predict  system  reliability 
before  development  begins.  This  is  accomplished  by  identifying  the  inher¬ 
ent  reliabilities  of  software  components,  combining  them  into  a  single 
prediction  of  software  reliability,  and  incorporating  this  prediction  into 
the  system  reliability  model.  Development  methodologies  can  then  be  chosen 
to  increase  overall  reliability,  and  cost  tradeoffs  can  be  performed  to 
determine  the  most  cost-effective  means  of  achieving  a  required  reliability 
In  the  event  that  desired  system  reliability  cannot  be  achieved  within 
budgetary  constraints,  information  obtained  from  the  methodology  will  be 
available  to  support  a  go  or  no-go  decision  at  a  time  prior  to  extensive 
investments  in  the  development. 

By  using  this  methodology  in  combination  with  other  reliability 
measurement  techniques,  the  original  estimates  of  inherent  component  reli¬ 
abilities  can  be  refined  as  those  components  become  testable.  The  method 
can,  therefore,  be  used  at  various  stages  of  the  development  process  to 
periodically  reevaluate  the  system  reliability  predictions  and  to  measure 
effectiveness  of  the  development  methods  employed. 

In  addition  to  being  recursive  and  reiterative,  the  method  is  designed 
for  expandability  and  flexibility.  There  are  no  constraints  on  the  number 
or  type  of  pi  factors  that  may  be  incorporated  into  a  particular  prediction 
As  the  state  of  the  art  of  software  development  advances,  it  will  be  an 
easy  task  to  incorporate  the  effects  of  additional  technical  and  managerial 
approaches . 

Many  software  development  experts  foresee  a  future  environment  of  off- 
the-shelf  or  standard  component  software  products  where  a  developer  will 
create  a  software  subsystem  by  logically  combining  existing  (high  reliabil¬ 
ity)  software  packages.  If  this  environment  is  realized,  the  inherent  com¬ 
ponent  reliabilities  essential  to  the  method  presented  here  will  be  readily 
available  and  highly  accurate.  Since  these  components  of  the  software 
subsystem  will  be  developed,  they  will  not  be  influenced  by  developing  pi 
factors  and  will  be  easily  inserted  into  the  calculations. 

Many  individual  factors  that  are  used  by  the  reliability  prediction 
methodology  will  require  refinement  as  the  state  of  the  art  advances.  The 
method  itself,  however,  is  sound,  practical,  flexible,  and  will  form  the 
basis  for  meaningful  prediction,  measurement,  and  estimation  of  operational 
system  reliability. 


6.0  DATA  COLLECTION 


6.1  Overview  and  Objective 

The  objective  of  this  phase  of  the  effort  was  to  collect  and  analyze 
data  from  existing  fielded  systems  and  compare  the  results  measured  with 
those  that  would  have  been  predicted  by  the  methodology. 

6.2  Approach 

The  approach  taken  was  to  initiate  a  broad  spectrum  data  collection 
effort  at  the  beginning  of  the  study;  to  determine  what  data  was  needed  as 
the  methodology  developed;  and  finally,  to  calculate  from  the  data  those 
parameters  that  directly  relate  to  the  inputs  and  predictions  produced  by 
the  method. 

Data  collection  involved  both  Martin  Marietta  sources  as  well  as 
external,  public  domain  data  bases.  Within  Orlando  Aerospace,  data  were 
collected  from  three  major  projects:  two  missile  systems,  and  one  command 
and  control  application.  Requests  were  sent  to  our  sister  divisions  in 
Baltimore  and  Denver  for  data  on  two  more  large-scale  Defense  projects. 
Three  major  data  bases  were  purchased  from  the  Data  and  Analysis  Center  for 
Software  (DACS). 

6.3  Results 


With  one  exception,  the  data  collection  effort  was  very  disappointing. 
The  data  available  was  either  incomplete  or  inconclusive.  Originally,  it 
was  thought  that  sufficient  data  could  be  gathered  within  our  own  organiza¬ 
tion.  In  fact,  the  best  source  located  was  data  gathered  from  our  Assault 
Breaker  project.  But  even  the  Assault  Breaker  data  is  insufficient  for 
validating  the  methodology  due  to  the  lack  of  detailed  operational  data. 

In  Volume  II  of  this  report,  the  Assault  Breaker  program  is  explained  in 
detail,  and  the  methodology  is  demonstrated  on  real  data.  Unfortunately, 
Assault  Breaker  performance  is  too  good.  Although  the  methodology  predicts 
a  very  high  reliability,  the  software  has  performed  even  better.  At  best, 
the  demonstration  establishes  credibility  of  the  prediction  technique. 

Essentially,  the  data  collected  was  unusable  for  one  or  more  of  the 
reasons  discussed  in  the  following  paragraphs. 

6.3.1  Fa i lure  Rate 


When  the  study  was  originally  proposed,  it  was  anticipated  that  a  cri- 
t’.cal  parameter  of  the  methodology  would  be  some  sort  of  software  failure 
rate.  So  many  models  have  been  developed  based  on  the  removal  of  errors 
over  time,  it  was  anticipated  that  our  methodology  would  have  a  similar 
dependency.  Chronological  failure  histories  of  some  of  our  internal  pro¬ 
jects  might  provide  such  information.  For  this  reason,  such  data  was 
collected  early  in  the  study  and  the  DACS  Software  Reliability  Dataset, 
compiled  by  John  Musa,  was  purchased.  As  the  methodology  evolved,  it 
became  clear  that  software  reliability  is  more  a  function  of  the  missions 
that  it  performs  than  the  amount  of  time  it  runs  successfully  between 
failures.  This  is  not  meant  to  imply  that  such  information  is  not 
meaningful . 
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RoLiabLlLty  measuremeuL  aiiii  estimaLioii  are  essential  to  evaluation  ot 
operational  systems.  The  methodology  developed  as  part  ot  this  study, 
however,  is  oriented  toward  prediction  at  a  time  when  no  failure  histories 
are  available.  Data  needed  to  adequately  validate  the  methodology  would 
consist  of  repetitious  executions  of  a  given  mission  scenario  witliin  a 
variety  of  input  domains.  When  a  failure  is  recorded,  the  fault  could  not 
be  repaired.  We  are  not  aware  that  such  an  experiment  has  been  accomplish¬ 
ed  . 

6.3.2  Latent  Defects 

Many  individuals  and  organizations  still  do  not  accept  the  notion  of 
software  reliability.  Because  software  does  not  exhibit  the  physical 
characteristics  of  hardware  devices,  many  professionals  devoutly  subscribe 
to  the  deterministic  view  of  software;  it  is  either  100  percent  or  zero 
percent  reliable.  In  reality,  this  is  a  true  assessment.  However,  it  is 
also  reality  that  software  failures  occur  statistically  in  a  manner  similar 
to  hardware  failures.  Despite  the  similarity,  software  faults  are  not 
typically  recorded  as  failures  for  reliability  measurement  purposes.  They 
are  treated  as  latent  defects,  fixed,  and  forgotten. 

b.3.3  Fault  Assessment 

Most  systems  that  have  been  operational  long  enough  to  have  established 
reliability  data  available  were  fielded  before  software  was  recognized  as  a 
separable  entity  of  the  system.  In  many  cases,  software  requirements  are 
specified  as  computer  requirements,  and  faults  uncovered  during  operation 
are  assessed  against  the  computer,  not  against  the  software.  Investigation 
into  specific  problem  descriptions  associated  with  operational  systems 
revealed  such  problem  descriptions  as:  "the  computer  went  down"  and  "the 
computer  miscalculated  the  coordinates."  The  failures  were  either  discounted 
when  the  computer  was  reinitialized  or  were  improperly  charged  against  the 
computer,  just  as  though  a  resistor  had  failed.  A  proper  evaluation  is 
possible  only  if  the  original  problem  reports  are  available.  It  is  vir¬ 
tually  impossible  to  use  data  base  information  or  summaries  to  determine 
software  performance  separate  from  computer  performance. 

6.3.4  Enhancements 

When  hardware  fails,  something  that  had  been  working  stops.  When 
software  fails,  the  fault  was  present  all  along.  When  hardware  fails,  it 
is  fixed  by  returning  it  to  its  previous  working  state.  When  software 
fails,  it  is  returned  to  a  better  state.  By  this  rationale,  many  software 
changes  are  not  recorded  as  fault  corrections,  but  rather  as  enhancements. 

\ga i n ,  detailed  analysis  of  individual  change  notices  reveals  which  changes 
were,  in  fact,  enhancements  and  which  ones  were  not.  However,  review  of 
large  scale  data  bases  is  inconclusive. 

6.3.5  Developer  versus  User 

When  a  company  such  as  Martin  Marietta  develops  and  delivers  a  defense 
system,  their  data  collection  terminates.  Although  extensive  records  are 
maintained  prior  to  delivery,  very  little  data  is  returned  after  delivery. 
Data  bases  available  from  developers,  therefore,  terminate  at  the  point 


where  the  system  became  operational.  Even  though  data  concerning  the  in¬ 
herent  factors  of  the  software  are  available  and  the  error  avoidance  and 
detection  mechanisms  used  during  development  are  known,  the  resulting  per¬ 
formance  data  are  not.  Conversely,  those  who  are  charged  with  maintaining 
a  fielded  system  have  performance  data,  but  they  do  not  have  the  develop¬ 
mental  information  needed  to  validate  a  prediction  methodology.  A  partic¬ 
ularly  frustrating  experience  is  to  have  an  abundance  of  operational  data 
and  an  abundance  of  developmental  data  for  different  systems. 

6.3.6  Military  versus  Commercial 

The  most  extensive  software  performance  data  bases  available  were 
compiled  for  commercial  systems.  Literally,  thousands  of  software  failures 
have  been  recorded  against  millions  of  hours  of  operation.  Reliability  of 
a  military  system  must  be  measured  against  its  operational  mission.  As 
defined  earlier,  system  software  reliability  is  the  probability  of  perform¬ 
ing  an  intended  mission  without  causing  system  outage  or  failure.  To 
record  a  software  failure  would,  therefore,  require  a  system  outage  or 
failure.  For  the  type  of  software  developed  for  weapon  systems,  an  outage 
and  a  failure  are  the  same.  A  missile  cannot  be  reinitialized  while  in 
flight.  Whereas,  most  commercial  applications  can  continue  to  be  used  with 
an  acceptable  failure  rate,  most  military  systems  require  a  very  high  prob¬ 
ability  of  working  correctly  every  time.  Defense  systems  data  bases, 
therefore,  tend  to  contain  extensive  data  on  mission  scenarios  rather  than 
to  meet  their  primary  purpose  (e.g.,  data  on  training  exercises  rather  than 
on  actual  flights) . 

6.4  Conclusions  and  Recommendations 

Although  the  data  collected  during  this  study  were  inconclusive  as  a 
method  validation  tool,  analysis  indicated  some  significant  shortcomings 
about  the  way  we  handle  software  data  throughout  its  life  cycle.  The 
recommendations  in  the  following  paragraphs  are  offered. 

6.4.1  Record  All  Software  Problems 

We  must  instill  in  data  collectors  and  recorders  the  importance  of 
properly  identifying  problem  sources.  In  most  cases,  the  personnel  charged 
with  operating  a  system  are  not  qualified  to  make  a  determination  of  the 
cause  of  a  problem.  When  a  system  defect  is  recorded,  it  must  eventually 
be  charged  against  some  aspect  of  the  system.  When  a  hardware  f.ailure  can¬ 
not  be  replicated,  the  failure  is  recorded  as  being  unverified  or  transient 
This  is  a  logical  and  well-accepted  practice.  However,  when  software 
fails,  a  fault  exists.  When  the  system  is  reinitialized  or  rerun,  we  are, 
in  most  cases,  changing  the  environment  (or  input  domain)  in  which  the 
software  was  operating  when  it  failed.  Sooner  or  later,  the  problem  will 
reoccur.  If  it  is  to  be  corrected,  it  is  essential  that  tlie  state  of  the 
system  during  the  problem  be  recorded  as  accurately  as  possible.  If  the 
system's  state  can  be  re-established,  the  fault  will  reraanitest  itself. 

All  software  errors  and  suspected  software  errors  must  be  documented,  even 
if  .1  simple  restart  makes  it  seem  to  disappear. 
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6.4.2  Identify  the  Source 


When  a  software  fault  is  isolated,  it  is  very  important  for  analysis 
to  reveal  the  exact  cause  of  the  problem.  There  is  no  such  thtn^;  as  an 
enhancement  resulting  from  a  system  failure.  Software  is  not  just  the  code 
that's  embedded  in  the  computer.  Software  is  also  the  design  logic  that 
resulted  in  the  code,  and  it  is  the  requirement  that  resulted  in  the  design 
11  It  IS  decided  to  enhance  the  software  to  extend  its  capabilities,  then 
the  original  capabilities  were  not  properly  stated  and  the  software  had  a 
requirements  deficiency.  Virtually  everyone  in  the  software  industry 
acknowledges  the  fact  that  incomplete,  ambiguous,  and  constantly  changing 
requirements  are  the  heaviest  contributors  to  software  costs  and  perform- 
mance  problems.  However,  we  deny  ourselves  the  data  needed  to  correct,  or 
at  least  minimize,  the  impact  by  labeling  requirement  and  design  changes  as 
enhancement  s . 

6.4.3  Distinguish  Hardware  From  Software 

The  physical  portion  of  a  computer  is  a  hardware  component  of  a  system 
The  logical  operation  of  that  computer  is  a  software  component  of  the 
system.  In  most  cases,  operations  personnel  cannot  distinguish  between  a 
computer  hardware  failure  and  a  computer  software  failure.  The  symptoms  in 
many  cases  are  the  same.  Recent  emphasis  on  automatic  fault  isolation  and 
built-in-test  is  encouraging.  As  mentioned  earlier,  it  is  essential  that 
software  faults  be  identified  as  such. 

6.4.4  Consistent  Data  Collection 

Procuring  agencies  interface  with  both  the  developer  of  a  system  and 
the  eventual  user  of  the  system.  They  alone  can  establish  and  influence  a 
meaningful  data  collection  effort  on  both  sides  of  the  delivery  milestone. 
The  developer  must  be  encouraged  or  paid  to  record  significant  facts  con¬ 
cerning  the  development  process.  Software  problem  reports  generated  during 
development  are  typically  devised  and  used  by  the  developer.  Since  the 
procuring  agency  is  not  generally  concerned  with  resolved  problems,  these 
reports  are  not  usually  identified  as  deliverable  data  items.  Likewise, 
the  techniques  used  by  the  contractor  during  development  are  usually  not 
documented  in  a  uniform  or  consistent  fashion.  They  are  hardly  ever  docu¬ 
mented  in  a  deliverable  data  item.  When  the  system  is  fielded,  the  oper¬ 
ational  user  begins  collecting  performance  data  without  the  benefit  of 
knowing  how  the  software  was  developed.  When  he  does  identify  a  problem, 
he  may  be  able  to  fix  it.  However,  he  cannot  determine  why  it  happened  or 
what  could  have  been  done  to  avoid  it . 

The  Software  Technology  for  Adaptable,  Reliable  Systems  (STARS!  pro¬ 
gram  has  recently  generated  a  series  of  Data  Item  Descriptions  that  provide 
a  uniform  and  consistent  format  for  recording  software  facts  during  and 
subsequent  to  development.  Although,  usage  of  the  forms  devised  will 
increase  the  cost  of  projects  using  them,  the  data  that  will  be  collected 
should  prove  invaluable  to  the  development  and  evaluation  of  new  software 
methods  and  techniques. 


7.0  CONCLUSIONS  AND  RECOMMENDATIONS 


The  methodology  derived  during  this  effort  should  be  applicable  to 
virtually  any  operational  system  which  employs  critical,  embedded  computers 
and  software.  The  method  is  both  mathematically  sound  and  computationally 
feasible  at  any  level  of  detail  required.  Like  all  reliability  models,  the 
accuracy  of  the  predictions  computed  is  highly  dependent  on  the  parameters 
used  during  the  calculations.  The  factors  used  in  hardware  reliability 

predictions  have  been  derived  and  measured  over  an  extensive  period  of 

time.  Until  recently,  very  little  quantitative  information  about  software 
factors  has  been  available.  The  true  value  of  the  methodology  will  not  be 

totally  realized  until  more  precise  measurement  of  software  factors  and 

characteristics  is  possible. 

With  the  increased  dependency  of  military  systems  on  embedded  software, 
the  industry  is  becoming  increasingly  more  interested  in  understanding, 
measuring  and  controlling  the  software  development  process  and  product. 

The  trend  toward  high-level  languages,  particularly  Ada,  the  trend  toward 
standard  computer  architectures,  and  the  enforcement  of  software  design  and 
development  standards  will  facilitate  the  derivation  and  use  of  additional 
measures  of  software  factors  which  can  be  used  directly  by  the  methodology. 

The  SAIC  study  [103]  mentioned  earlier  will  be  concluded  soon.  The 
measures  defined  therein  appear  to  be  both  meaningful  and  compatible  with 
the  methodology  developed  during  this  effort.  An  effort  is  needed  to 
directly  validate  the  feasibility,  accuracy  and  cost  effectiveness  of  both 
the  SAIC  measures  and  the  Martin  Marietta  methodology.  It  is  strongly 
recommended  that  consideration  be  given  to  the  definition  of  a  follow-on 
effort  to  validate  the  study  results  on  an  actual  military  system  develop¬ 
ment  contract.  The  effort  should  be  accomplished  concurrently  with  the 
system  being  developed.  Data  concerning  the  development  process  itself 
should  be  captured  and  used  in  the  prediction  methodology.  The  SAIC 
measurements  should  be  accomplished  at  various  stages  during  the  system 
development  and  detected  faults  should  be  recorded  and  analyzed  when  they 
happen,  not  after  it  is  too  late  to  determine  their  root  causes.  Such  an 
effort  would  produce  a  wealth  of  data  not  currently  available  and  would 
form  an  intelligence  base  usable  by  Air  Force  planners  and  civilian  con¬ 
tractors  to  determine  the  most  cost  effective  means  of  developing  high 
reliability  systems. 

The  method  developed  during  this  effort,  much  like  the  corresponding 
hardware  methodology  described  in  MIL-HDBK-21 7D,  is  applied  by  defining  the 
individual  parts  (modules)  which  make  up  the  system  and  by  defining  the 
interactions  between  them.  Although  the  method  can  be  applied  manually,  it 
lends  itself  quite  well  to  computerization.  Furtlierraore ,  one  of  its  in¬ 
tended  uses  IS  as  a  tool  for  performing  tradeoff  analyses  of  various  design 
approaches  and  techniques.  It  is  suggested  that  consideration  be  given  to 
the  construction  of  a  user-friendly,  conversational  and  flexible  computer 
program  which  would  lead  a  reliability  engineer  through  the  method.  Such  a 
program  could  be  designed  around  the  current  method  using  current  parameters 
and  factors.  It  could  be  designed  for  expansion  and/or  modification  as  the 
techniques  and  metrics  evolve.  Furthermore,  it  could  be  extremely  valuable 
as  a  tool  to  begin  the  accumulation  and  refinement  of  statistical  data 
bases  needed  for  accurate  software  reliability  measurement  and  prediction. 
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The  Ada  programming  language  will  very  shortly  be  the  source  language 
for  all  DoD-embedded ,  mission-critical  software.  It  would  be  most  appropri 
ate  to  initiate  data  collection  concerning  the  effects  of  Ada  on  software 
and  system  reliability.  A  study  of  the  language  itself  and  an  expansion  of 
the  combined  system  reliability  prediction  methodology  to  accommodate  it 
would  be  a  valuable  asset  to  Air  Force  System  planners  and  project  officers 
as  they  begin  and  carry  out  the  transition  to  Ada. 
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ACCEPTANCE  CRITERIA  --  The  criteiia  a  software  product  must  meet  to  success 
fully  complete  a  test  phase  or  meet  delivery  requirements. 

ACCEPTANCE  TESTING  —  Formal  testing  conducted  to  determine  whether  a  system 
satisfies  its  acceptance  criteria  and  to  enable  the  customer  to  determine 
whether  to  accept  the  system.  See  also  QUALIFICATION  TESTING. 

AUTOMATED  DESIGN  TOOL  —  A  software  tool  which  aids  in  the  synthesis,  ana¬ 
lysis,  modeling  or  documentation  of  a  software  design.  Examples  include 
simulators,  analytic  aids,  design  representation  processors  and  documen¬ 
tation  generators. 

AVAILABILITY  —  The  probability  that  computer  software  is  capable  of  fuuac- 
tioning  in  accordance  with  requirements  at  any  time.  This  probability  is 
often  measured  with  respect  to  total  need  time. 

BATCH  PROCESSING  —  A  technique  by  which  items  are  coded  and  collected  into 
groups  for  processing. 

CDR  —  See  CRITICAL  DESIGN  REVIEW. 

CERTIFICATION  —  The  process  of  confirming  that  a  system  is  operationally 
effective  and  capable  of  satisfying  mission  requirements  under  realistic 
operating  conditions. 

CHANGE  REQUEST  —  See  SOFTWARE  CHANGE  REQUEST. 

CLARITY  —  The  ability  of  the  computer  program  to  be  easily  understood.  It 
is  a  measure  not  only  of  the  computer  program  itself,  but  also  of  its  sup¬ 
porting  documentation. 

COMPATIBLE  HAREWARE/30FTWARE  PREDICTION  MODEL  —  Suitable  interpretation  of 
hardware  and  software  mathematical  relationships  for  combined  computations 
so  as  to  make  feasible  prediction  of  the  System  Reliability. 

COMPLEXITY  —  The  degree  of  complication  of  a  system  or  system  component, 
determined  by  such  factors  as  the  number  and  intricacy  of  interfaces,  the 
number  and  intricacy  of  conditional  branches,  the  degree  of  nesting,  the 
types  of  data  structures,  and  other  system  characteristics. 

CONFIGURATION  CONTROL  —  The  systematic  evaluation,  coordination,  approval  or 
disapproval,  and  implementation  of  all  approved  changes  in  the  configuration 
of  a  configuration  item  after  formal  establishment  of  its  approved  technical 
documentation. 

CONFIGURATION  CONTROL  BOARD  —  The  authority  responsible  for  evaluating  and 
approving  or  disapproving  proposed  engineering  changes,  and  ensuring  imple¬ 
mentation  of  the  approved  changes. 


CONFIGURATION  ITEM  —  An  aggregation  of  hardware/computer  software,  or  any  of 
its  discrete  portions,  that  satisfy  an  end-use  function  and  are  designated 
by  the  Government  for  configuration  management. 

CONFIGURATION  MANAGEMENT  —  A  discipline  applying  technical  and  administrative 
direction  and  surveillance  to  identify  and  document  the  functional  and 
physical  characteristics  of  a  configuration  item,  control  changes  to  those 
characteristics,  record  and  report  change  processing  and  implementation 
status,  and  verify  compliance  with  specification  and  other  related  contract 
requirements. 

CONTROL  VARIABLES  —  Dynamic  program  data  which  affects  or  controls  processes 
within  other  modules  or  subprograms. 

CORRECTNESS  —  The  ability  of  the  computer  program  to  perform  exactly  and 
correctly  all  of  the  functions  required  by  the  specifications. 

COUPLING  —  See  DATA  COUPLING  and  LOGICAL  COUPLING 

CRITICAL  DESIGN  REVIEW  (CDR)  —  A  formal  technical  design  review  conducted  to 
ensure  that  the  detailed  design  satisfies  the  requirements  correctly  and 
completely.  It  is  conducted  after  completion  of  the  detailed  design  but 
prior  to  coding.  It  establishes  the  design  baseline. 

DATA  COUPLING  —  An  inter-relationship  between  or  among  program  modules  in 
which  data  items  are  shared  without  formal  parameter  passing. 

DECISION  TABLE  —  A  table  of  all  conditions  that  are  to  be  considered  in  the 
description  of  a  problem,  together  with  the  actions  to  be  taken.  The  two 
are  linked  by  "decision  rules"  vdiich  tie  each  combination  of  conditions 
with  a  corresponding  combination  of  actions. 

DESIGN  FACTORS  —  Factors  which  can  be  characterized  as  reliability  design 
tools  or  methodologies  (e.g.,  top-down  design,  modularity,  structured 
programming,  etc.). 

DETAILED  DESIGN  SPECIFICATION  —  This  document  provides  complete  programming 
design  sufficiently  detailed  for  a  programmer  to  code  from  with  minimal 
additional  direction. 

DEVELOPMENT  FACTORS  —  Factors  which  can  be  characterized  as  being  part  of  the 
software  relieibility  engineering  development  process  (e.g.,  test-debug-fix, 
use  of  developmental  aids  or  standards,  quality  assurance  measures,  etc.). 


DUTY  CYCLE  —  A  measure  of  the  need  time  of  a  computer  program  or  portion  of 
the  program  with  respect  to  total  system  time. 


ECONOMY  —  See  EFFICIENCY. 
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EFFICIE3VICY  —  A  measure  of  the  use  of  high-performance  algorithms  and  conser¬ 
vation  of  use  of  resources  to  minimize  the  cost  of  computer  operation. 
Sometimes  referred  to  as  ECONOMY. 

EMBEDDED  SOFTWARE  —  An  interactive  assembly  of  computer  programs  and  conputer 
data  that  is  integral  to  a  major  system  whose  primary  function  is  not  data 
processing. 

ENGINEERING  CHANGE  NOTICE  —  A  document  used  to  process  changes  to  baseline 
documents  and  vdiich  includes  both  notice  of  an  engineering  change  to  a 
configuration  item  and  the  supporting  documentation  by  which  the  change  is 
described. 

FAILURE  —  See  SOFTWARE  FAILURE. 

FAULT  —  A  software  defect  that  causes  program  operation  to  fail  to  perform 
program  requirements. 

FAULT  AVOIDANCE  —  The  act  of  eliminating  the  mechanisms  which  cause  erroneous 
software  to  be  created  and/or  the  application  of  mechanisms  which  encourage 
or  support  correct  software  creation,  it  relates  to  the  elimination  of 
errors  before  they  occur. 

FAULT  CORRECTICW  —  The  act  of  removing,  avoiding  or  otherwise  negating  the 
effects  of  a  detected  fault.  Cein  be  accomplished  automatically  by  the 
software  or  by  alteration  of  the  software. 

FAULT  DETECTION  —  The  recognition  of  the  presence  of  a  software  defect 
either  by  its  external  manifestations  or  by  an  inspection  of  the  software 
itself.  Inplied  in  the  definition  is  the  ability  to  locate  and/or  isolate 
the  defect  itself. 

FAULT  TOLERANCE  —  The  6±)ility  of  the  computer  program  to  perform  correctly 
despite  the  presence  of  error  conditions. 

FCA  ~  See  FUNCTIONAL  OONFIGURATON  AUDIT. 

FLEXIBILITY  —  A  measure  of  the  extent  to  which  the  computer  program's  design 
allows  it  to  perform  or  to  be  easily  modified  to  perform  functions  beyond 
the  scope  of  its  original  requirements. 

FLOW  CHART  —  A  symbolic  representation  of  the  functional  flow  and  eui  abbre¬ 
viated  description  of  the  inputs,  processing,  outputs  and  flow  of  control 
of  a  computer  program  or  portion  of  the  program. 

FORMAL  QUALIFICATION  TESTING  (FQT)  —  Testing  conducted  prior  to  Functional 
Configuration  Audit  to  demonstrate  CPCI  compliance  with  all  applicable 
software  specifications. 

FQT  ~  See  FORMAL  QUALIFICATION  TESTING 


FUNCTIONAL  CONFIGURATION  AUDIT  (FCA)  --  Audit  to  verify  that  the  actual  per¬ 
formance  of  the  configuration  items  complies  with  the  B-5  development 
specifications, 

FUNCTICWAL  DECOMPOSITI(»J  —  A  method  of  designing  a  system  by  breaking  it  down 
into  its  conponents  in  such  a  way  that  the  components  correspond  directly 
to  system  functions  and  subfunctions. 

FUNCTIC»IAL  DESIGN  SPECIFICATIW  —  This  document  establishes  the  functional 
design  of  the  software  at  the  computer  program  level;  provides  sufficient 
design  information  to  acconplish  the  goals  of  the  Preliminary  Design  Review. 

GENERALITY  —  The  ability  of  the  conputer  program  to  perform  its  intended 
functions  over  a  wide  range  of  usage  modes  and  inputs,  even  when  not  di¬ 
rectly  specified  as  a  requirement. 

HARDWARE  RELIABILITY  —  The  probability  that  the  required  hardware  in  a  system 
will  operate  failure  free  in  a  specified  environment  for  the  prescribed 
missions  and  time  periods. 

HIERARCHICAL  CCKTROL  —  A  sequence  of  control  which  consists  of  multiple 
levels  of  decomposition,  general  to  specific. 

HIERARCHICAL  DESIGN  —  A  design  which  consists  of  multiple  levels  of  decom¬ 
position,  general  to  specific. 

HIERARCHICAL  I^IPUT-PRCCESSI^K>-OUTPUT  (HIPO)  CHARTS  —  A  document  which  con¬ 
sists  of  diagrams  illustrating  the  functional  flow  of  inputs,  processing 
and  outputs  on  multiple  levels  of  decomposition,  general  to  specific. 

HIGH  ORDEH  LANGUAGE  —  A  programming  language  v^iich  provides  compression  of 
computer  instructions  such  that  one  program  statement  represents  meuny 
machine  language  instructions.  It  is  nonproblem  specific  and  is  used  by 
programmers  to  communicate  with  the  computer. 

HIPO  CHARTS  —  See  HIERARCHICAL  INPUT-PROCESSING-OUTPUT  CHARTS 

INDEPENDENT  VERIFICATION  AND  VALIDATION  (IV&V)  —  Verification  and  Validation 
of  a  software  product  is  performed  by  an  organization  that  is  both  tech¬ 
nically  and  managerially  separate  from  the  organization  responsible  for 
developing  the  product. 

INTERFACE  DESICJJ  SPECIFICATION  —  This  is  an  optional  document  which  is  re¬ 
quired  whenever  the  system  contains  two  or  more  computers  that  must  com¬ 
municate  with  each  other.  It  provides  a  detailed  logical  description  of 
all  data  units,  messages,  control  signals  and  communication  conventions 
between  the  digital  processors. 

INTEROPERABILITY  —  A  measure  of  the  ease  by  which  a  computer  program  cauti  be 
made  to  interface  with  other  computer  programs. 
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INTRINSIC  FACTORS  —  Factors  v^ich  can  be  termed  as  inherent  characteristics 
or  attributes  of  the  software  (e.g.,  language,  conplexity,  size,  etc.). 

IV&V  —  See  INDEPENDENT  VERIFICATION  AND  VALIDATION. 

LOGICAL  COUPLING  —  This  is  the  relationship  which  exits  between  program  mod¬ 
ules  due  to  the  passage  of  control  variables  from  one  module  to  the  other. 
Whereas  data  variables  provide  parameters  to  and  from  a  module,  control 
variables  affect  the  logical  operation  of  the  module  itself.  See  also 
DATA  COUPLING. 

MAINTAINABILITY  —  A  characteristic  of  design  and  installation  which  is  ex¬ 
pressed  as  the  probability  that  an  item  will  be  retained  in  or  restored  to 
a  specified  condition  within  a  given  period  of  time,  when  maintenance  is 
performed  in  accordance  with  prescribed  procedures  eind  resources;  i.e.,  a 
measure  of  the  extent  to  which  the  computer  program  can  be  easily  altered 
or  expanded  to  satisfy  new  requirements  or  to  correct  deficiencies. 

MODIFIABILITY  —  The  characteristics  of  being  easy  to  modify;  one  aspect  of 
maintainability.  This  inplies  controlled  change  in  which  some  parts  or  as¬ 
pects  remain  the  same  v^ile  others  are  altered  in  such  a  way  that  a  desired 
new  result  is  obtained.  This  measurement  includes  consideration  of  the 
extent  to  which  likely  candidates  for  change  are  isolated  from  the  rest  of 
the  computer  program. 

MODULAR  CONSTRUCTICN  —  An  organization  of  the  functions  of  the  computer  pro¬ 
gram  into  a  set  of  discrete  progreim  modules, 

MODULAR  DECOMPOSITION  —  See  FUNCTIONAL  DECOMPOSITION. 

MODULARITY  —  The  extent  to  which  the  computer  program  is  segmented  into 
single-purpose,  single-entry,  single-exit  modules. 

MODULE  —  A  discrete  identifiable  set  of  computer  instructions  usually  handled 
as  a  unit  by  assembler,  conpiler,  linkage  editor,  loading  routine,  or  other 
type  of  routine  or  subroutine.  It  is  frequently  defined  as  the  lowest  stand 
alone,  teste±»le  set  of  instructions. 

MODULE  COUPLING  —  See  DATA  COUPLING  and  LOGICAL  COUPLING 

OPERATING  SYSTEM  —  Software  that  controls  the  execution  of  programs.  An  op¬ 
erating  system  may  provide  services  such  as  resources  allocation  scheduling, 
input/output  control,  eind  data  management.  Although  operating  systems  are 
predominantly  software,  partial  or  conplete  hardware  implementations  are 
possible.  An  operating  system  provides  support  in  a  single  spot  rather 
than  forcing  each  program  to  be  concerned  with  controlling  hardware.  See 
also  SYSTEM  SOFTWARE.  * 

OPERATIONAL  FACTORS  —  Factors  which  can  be  derived  and  characterized  from 
system  requirement  and  specification  documents  (e.g.,  operational/lmission 
scenarios,  inputs-outputs  functions,  performance  criteria,  etc.). 


PCA  --  See  PRELIMINARY  CONFIGURATION  AUDIT. 


PDL  —  See  PROGRAM  DESIO^  LANGUAGE. 

PDR  —  See  PRELIMINARY  DESIGN  REVIEW. 

PHYSICAL  CONFIGURATION  AUDIT  (PCA)  —  A  formal  examination  of  the  as-built 
version  of  a  configuration  item  against  its  technical  documentation  to  en¬ 
sure  the  adequacy,  completeness,  and  accuracy  of  the  technical  design 
documentation. 

PQT  —  See  PRELIMINARY  QUALIFICATION  TESTING. 

EKIRTABILITY  —  The  characteristic  of  computer  software  which  allows  it  to  be 
used  in  a  computer  environment  different  from  the  one  for  which  it  was 
originally  designed. 

PRELIMINARY  DESIGN  REVIEW  (PDR)  —  A  formal  technical  review  of  the  basic 
design  approach.  It  is  held  after  the  completion  of  preliminary  design 
efforts  but  prior  to  the  start  of  detailed  design.  See  also  SYSTEM  DESIGN 
REVIEW  and  CRITICAL  DESIO^  REVIEW. 

PRELIMINARY  QUALIFICATION  TESTING  (PQT)  —  An  incremental  process  which  pro¬ 
vides  visibility  and  control  of  the  computer  program  development  during  the 
time  period  between  the  Critical  Design  Review  (CDR)  and  Formal  Qiaalifi- 
cation  Testing  (FQT);  conducted  for  those  functions  critical  to  the  CPCI. 

PROGRAM  —  All  the  software  that  can  physically  interrelate  as  an  entity.  It 
is  also  the  neune  given  to  the  highest  level  function  in  a  hierarchical 
design. 

PROGRAM  DESIGN  LANGUAGE  (PDL)  —  A  language  with  special  constructs  and, 
sometimes,  verification  protocols  used  to  develop,  analyze,  and  document 
a  design. 

PROGRAM  SPECIFICATION  LANGUAGE  (PSD  —  A  language  used  to  specify  the  design, 
requirements,  behavior,  or  other  characteristics  of  a  system  or  system  com¬ 
ponent  . 

PROGRAM  SIZE  —  A  measurement  of  size  usually  expressed  as  the  number  of  lines 
of  code,  the  number  of  computer  instructions  or  the  number  of  bytes  of  code. 

PROGRAM  SUPPORT  ENVIRONMENT  —  An  integrated  collection  of  tools  accessed  via 
a  single  commemd  language  to  provide  programming  support  capabilities 
throughout  the  software  life  cycle.  The  environment  typically  contains 
tools  for  designing,  editing,  compiling,  loading,  testing,  configuration 
management  and  project  management. 

PSEUDO  CODE  —  A  combination  of  programming  language  and  natural  language  used 
for  computer  program  design. 
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PSL  —  See  PROGRAM  SPECIFICATION  LANGUAGE. 

QUALIFICATION  TESTING  —  Formal  testing,  usually  conducted  by  the  developer 
for  the  customer,  to  demonstrate  that  the  software  meets  its  specified  re¬ 
quirements.  See  also  ACCEPTANCE  TESTING,  PRELIMINARY  QUALIFICATION  TESTING, 
AND  FORMAL  QUALIFICATICW  TESTING. 

QUALITY  ASSURANCE  —  A  planned  and  systematic  pattern  of  all  actions  necessary 
to  provide  adequate  confidence  that  the  item  or  product  conforms  to  estab¬ 
lished  technical  requirements. 

READABILITY  —  A  measure  of  how  well  a  skilled  programmer  who  was  not  the 
original  creator  of  the  computer  program  can  understand  the  program  and 
correlate  it  to  the  original  and  to  new  requirements. 

REAL-TIME  PROCESSING  —  The  processing  of  information  or  data  in  a  manner 
sufficiently  rapid  that  the  results  of  the  processing  are  available  in  time 
to  influence  the  process  being  monitored  or  controlled. 

REPAIRABILITY  —  The  extent  to  which  a  change  to  correct  a  deficiency  can  be 
localized,  so  as  to  have  minimal  influence  on  other  program  modules,  logic 
paths  or  documentation. 

REQUIREMENTS  SPECIFICATION  —  A  specification  that  sets  forth  the  requirements 
for  a  system  or  a  system  component;  for  example,  a  software  configuration 
item.  Typically  included  are  functional  requirements,  performance  require¬ 
ments,  interface  requirements,  design  requirements  and  development  stan¬ 
dards. 

REQUIREMENTS  TRACEABILITY  MATRIX  —  A  set  of  tables  that  provides  traceability 
of  software  requirements  from  the  system  specification  to  the  individual 
item  requirements  specifications,  to  the  design  specification  which  imple¬ 
ments  the  requirements,  and  to  the  software  plans  and  procedures  that  verify 
that  requirements  have  been  fully  inplemented. 

RESILIENCE  —  A  measure  of  the  computer  program's  ability  to  perform  in  a 
reasonable  manner  despite  violations  of  the  assumed  usage  and  input  conven¬ 
tions.  Also  referred  to  as  ROBUSTNESS. 

REUSABILITY  —  A  measure  of  the  extent  to  which  the  computer  program  can  be 
used  in  an  application  different  from  the  one  for  which  it  was  developed. 

REVIEWS  —  See  specific  entries,  e.g.,  SDR,  PDR,  CDR. 

ROBUSTNESS  —  See  RESILIENCE. 

SDR  ~  See  SYSTEM  DESIGN  REVIEW. 

SELF-TEST  CAPABILITY  —  The  extent  to  which  the  computer  program  can  be  easily 
and  thoroughly  tested  by  internal  procedures. 


SOFTWARE  CHANGE  REQUEST  —  A  document  used  to  describe  and  process  proposed 
changes  to  baseline  software  and  its  associated  documentation. 

SOFTWARE  CC*4FIGURATION  MANAGEMENT  PLAN  —  This  document  describes  the  con¬ 
tractor's  organization  for  configuration  management,  the  procedures  that 
will  be  used  to  inclement  tailored  contractural  requirements,  and  the  per¬ 
sons/groups  responsible  for  each  particular  phase  of  configuration  manage¬ 
ment. 

SOFTWARE  CORRECTIVE  MODIFICATION  —  The  periodic  updating  of  the  software  to 
preclude  system  failure  when  processing  potential  data  sets. 

SOFTWARE  DESIGN  SPECIFICATION  —  This  document  describes  the  assignment  of 
each  of  the  software  requirements  to  a  specific  functional  software  module, 
the  functional  interface  for  each  module,  the  data  base  utilized  by  each 
module,  cind  the  design  implementation  which  has  been  built  into  the  oper¬ 
ational  software. 

SOFTWARE  DEVELOPMENT  LIBRARY  —  The  libraries,  library  procedures  and  auto¬ 
mated  aids  used  to  maintain  control  of  the  software  baseline  by  providing 
a  consistent,  systematic  and  orderly  method  for  organizing,  maintaining  auvJ 
controlling  a  project's  computer  program  elements  eind  documentation  during 
the  developnent  phase. 

SOFTWARE  DEVELOPMENT  PLAN  —  This  document  presents  the  comprehensive  pleui  for 
the  project's  software  development  activities  by  describing  the  software 
development  organization,  the  software  design  and  testing  approach,  the 
programs  atfid  documentation  that  will  be  produced,  software  milestones  euid 
schedules,  and  the  allocation  of  develojxnent  resources. 

SOFTWARE  FAILURE  —  The  inability  to  perform  an  intended  logical  operation  in 
the  presence  of  the  specified  data/environment  due  to  a  fault  in  the  soft¬ 
ware. 

SOFTWARE  MAINTAINABILITY  MODEL  —  A  mathematical  model  that  may  be  derived 
from  prior  experience  in  correcting  software  faults  that  predicts  frequency 
of  faults  of  various  categories,  and  may  include  suitable  parameters  to 
accomodate  results  of  timeline  analysis  of  software  corrective  and  preven¬ 
tative  maintenance,  determination  of  mean-time-to-restore  (MTTR)  as  well 
as  maximum  restore  time  for  the  required  percentile  of  the  timeline  data, 
and  determination  of  optimum  performance  of  software  corrective  and  preven¬ 
tative  modification  tasks,  including  frequency  eund  duration. 

SOFTWARE  PROBLEM  REPORT  —  A  report  of  a  program  defficiency  identified 
during  software  qualification,  test,  system  integration  and  test,  or  sys¬ 
tem  operation,  which  appears  to  be  software  related. 

SOFTWARE  QUALITY  ASSURANCE  PLAN  —  This  document  is  produced  during  the  soft¬ 
ware  planning  phase  and  describes  the  procedures  that  will  be  used  to  im¬ 
plement  Software  Quality  Assurance  control,  and  the  persons/groups  respon¬ 
sible  for  each  phase  of  Software  Quality  Assurance. 


SOFTWARE  RELIABILITY  —  The  probability  that  the  required  software  of  a  sys¬ 
tem  will  perform  its  intended  functions  for  the  prescribed  missions  and 
time  periods  in  the  specified  operating  environment  without  causing  system 
outage  or  failure. 

SOFTWARE  RELIABILITY  PREDICTION  MODEL  —  A  mathematical  model  that  could  in¬ 
clude  appropriate  parameters  such  as  code  complexity,  branching  numerics, 
structured/modular  format  utilization,  execution  rate,  timing  restrictions, 
and  data  con¥)lexity,  predictability  and  variability,  as  may  be  verified  by 
test  data. 

SOFTWARE  REQUIREMENTS  REVIEW  (SRR)  —  A  review  to  achieve  formal  agreement 
between  the  customer  and  the  developer  that  the  software  requirements  spec¬ 
ifications  are  complete  and  accurate. 


SOFTWARE  REQUIREMENTS  SPECIFICATION  —  This  document  establishes  the  require¬ 
ments  for  the  performance,  design,  test  and  qualification  of  the  conputer 
program.  See  also  REQUIREMENTS  SPECIFICATION. 

SOFTWARE  SUPPORT  LIBRARY  —  A  software  library  containing  computer  readable 
and  human  readable  information  relevant  to  a  software  development  effort. 

SOURCE  LISTING  —  A  document  that  displays  tabulated  data  identifying  the 
sequential  appearance  of  instructions  as  they  appear  on  the  computer  pro¬ 
gram  media. 

SRR  —  See  SOFTWARE  REQUIREMENTS  REVIEW. 

SPECIFICATION  CHANGE  NOTICE  —  A  formal  notification  of  a  change  in  the  spec¬ 
ification. 

STEPWISE  REFINEMENT  —  The  process  whereby  steps  are  taken  in  the  following 
order:  (1)  the  total  concept  is  formulated  ,  (2)  the  functional  specifica¬ 
tion  is  designed,  (3)  the  functional  specification  is  refined  at  each  inter¬ 
mediate  step  where  intermediate  steps  include  code  or  processes  required  by 
the  previous  step,  and  (4)  final  refinements  are  made  to  completely  define 
the  problem. 

STRUCTURE  CHART  —  A  design  and  documentation  techinique  used  in  structured 
programming  to  show  the  purpose  and  relationships  of  the  various  modules 
in  a  proqram. 

STRUCTURED  APPROACH  —  An  approach  to  software  design  which  consists  of  using 
stepwise  refinement  to  formulate  and  define  a  problem.  See  also  STRUCTURED 
DESIGN. 

STRUCTURED  CODE  —  Code  that  has  been  generated  with  a  limited  number  of  well- 
defined  control  structures  using  stepwise  refinement.  See  also  STRUCTURED 
PROGRAMMING. 
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STRUCTORED  DESIC24  —  A  disciplined  approach  to  software  design  which  adheres 
to  a  specified  set  of  rules  based  on  principles  such  as  top-down  design, 
stepwise  refinement  and  data  flow  analysis. 

STRUCTURED  PROGRAMMING  —  A  computer  program  constructed  of  a  basic  set  of 
control  structures,  each  one  having  one  entry  point  and  one  exit.  The  set 
of  control  structures  typically  include;  sequence  of  two  or  more  instruc¬ 
tions,  conditional  selection  of  one  of  two  or  more  instructions  or  sequence 
of  instructions,  and  repetition  of  an  instruction  or  a  sequence  of  instruc¬ 
tions. 

SUBPROGRAM  —  A  confute r  program  that  can  be  part  of  another  program. 

SYSTEM  AVAILABLILITY  —  The  probability  or  proportion  of  operational  time 
that  the  hardware  and  software  is  in  the  required  operable  eind  comraiteible 
state  at  a  time  when  the  mission  is  required  with  a  specified  data  environ¬ 
ment. 

SYSTEM  CAPABILITY  —  The  probability  that  the  hardware  and  software  cewi  a- 
chieve  the  required  mission  objectives  given  the  operational  conditions, 
including  data  environment,  during  the  mission. 

SYSTEM  DEPENDABILITY  —  The  probability  that  the  hardware  and  software  will 
perform  successfully  during  one  or  more  required  sequences  of  a  mission, 
given  the  hardware  and  software  status  at  the  start  of  the  mission  (avail¬ 
ability)  . 

SYSTEM  DESIGN  REVIEW  (SDR)  —  A  formal  review  conducted  to  evaluate  the  oj>- 
timization,  traceability,  correlation,  completeness  and  risk  of  the  allo¬ 
cation  of  system  requirements  among  systenf  components  including  software. 

SYSTEM  EFFECTIVENESS  —  The  measure  of  the  degree  to  which  the  hardware  and 
software  achieve  the  mission  requirements  in  the  operational  environment 
as  evidenced  in  system  availability,  dependability  and  capability. 

SYSTEM  EFFECTIVENESS  MODEL  —  A  mathematical  model  encompassing  both  hardware 
cind  software  for  a  prior  prediction,  a  pre-operational  test  evaluation  or 
an  operational  demonstration  of  the  deliverable  system  effectiveness.  The 
model  should  encompass  the  foregoing  defined  parameters  and  include  a  prac¬ 
tical  means  of  computation  and  analysis.  Implementation  of  the  model  is 
generally  demonstrated  with  data  from  other  programs  or  data  assumed  from 
requirements,  prior  to  application  in  the  current  program. 

SYSTEM  INTEGWiTlON  TESTING  —  The  process  of  testing  an  integrated  hardware 
and  software  system  to  verify  that  the  system  meets  its  specified  require¬ 
ments. 

SYSTEM  REQUIREMENTS  SPECIFICATION  —  This  document  states  the  technical  and 
mission  requirements  for  a  system  as  an  entity,  allocates  requirements  to 
functional  areas,  and  defines  the  interfaces  between  or  among  the  functional 


TEST  ORGANIZATION  —  A  group  responsible  for  preparing  test  plans  and  proce¬ 
dures,  executing  the  test  procedures,  and  analyzing  the  test  results  in 
order  to  verify  that  the  system  performed  its  intended  functions.  This  group 
is  also  responsible  for  documenting  problems  detected  during  testing  and 
verifying  by  retest  that  corrections  to  the  software  work  properly. 

TEST  PLANS  AND  PROCEDURES  —  Documents  which  set  forth  how  the  system  or  con¬ 
figuration  item  will  be  qualified,  describing  on  the  system  level  how  it 
will  be  demonstrated  that  each  performance  requirement  stated  in  the  System 
Functional  Requirements  Specification  has  been  met.  Similarly  at  the  con¬ 
figuration  item  level,  these  documents  describe  the  method  of  qualifying  a 
configuration  item  against  functional  requirements  stated  in  its  Software 
Functional  Requirements  Specification. 

TEST  READINESS  REVIEW  (TRR)  —  A  review  conducted  prior  to  each  test  to  ensure 
adequacy  of  the  documentation  and  to  establish  a  configuration  baseline. 

TESTABILITY  —  The  extent  to  which  a  software  product  assists  in  the  estab¬ 
lishment  of  verification  criteria  and  supports  evaluation  of  its  perfor¬ 
mance. 

TOP-IXWN  APPROACH  —  An  approach  vdiich  identifies  major  functions  to  be  ac¬ 
complished,  then  proceeds  from  there  to  an  identification  of  the  lesser 
functions  that  derive  from  the  major  ones. 

TOP-DOWN  DESIGN  —  An  ordering  to  the  sequence  of  decisions  which  are  made  in 
the  decomposition  of  a  software  system  by  beginning  with  a  simple  descrip¬ 
tion  of  the  entire  process  (top  level).  Through  a  succession  of  refinements 
of  what  has  been  defined  at  each  level,  lower  levels  are  specified. 

TRR  —  See  TEST  READINESS  REVIEW 

UNIT  DEVELOPMENT  FOLDER  —  This  is  a  software  development  notebook  that  is 
used  to  collect  and  organize  software  requirements  and  products  for  a  unit 
of  software  as  they  are  produced.  It  contains  all  requirements  documen¬ 
tation,  flow  charts,  discrepancy  reports  and  test  results. 

UNIT  LEVEL  TESTING  —  Testing  to  verify  program  unit  logic,  computational 
adequacy,  data  handling  capability,  interfaces  and  design  extremes,  and 
to  execute  and  verify  every  branch. 

USABILITY  —  The  characteristic  of  software  which  is  indicative  of  its  respon¬ 
siveness  to  human  factors  considerations.  It  is  a  measure  of  how  well  the 
software  has  utilized  natural  and  convenient  techniques  for  human  operation. 

VALIDATION  —  The  evaluation,  integration  and  test  activities  carried  out  at 
the  system  level  to  ensure  that  the  system  satisfies  the  performance  and 
design  criteria  in  the  system  specification. 
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VALIDITY  —  The  ability  of  the  computer  program  to  provide  the  perfornance, 
functions  and  interfaces  that  are  sufficient  for  beneficial  application  in 
the  intended  user  environment.  Validity  pertains  to  the  specifications  as 
well  as  the  resulting  software. 

VERIFICATION  —  The  interactive  process  of  determining  vdiether  the  product  of 
each  step  of  the  configuration  item  development  process  fulfills  all  of  the 
requirements  levied  by  the  previous  step. 

VERSION  DESCRIPTION  DOCUMENT  —  This  document  provides  a  file  that  indicates 
the  exact  information  on  the  software  media  and  accompanies  a  configuration 
item  to  provide  pertinent  data  regarding  the  computer  program. 

WALKTHROUGH  —  A  process  by  which  a  team  of  programming  personnel  do  eun  in- 
depth  review  of  a  program  or  portion  of  a  program  by  inspection  to  detect 
errors  and  improve  program  reliability. 
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1)  TITLE:  SOFTWARE  RELIABILITY  MODELLING  AND  ESTIMATION  TECHNIQUES 

AUTHOR:  AMRIT  L.  GOEL,  SYRACUSE  UNIVERSITY 

DOC_DATE:  82/10 

SOURCE:  RADC 

ABSTRACT:  This  report  presents  the  results  of  the  software  reliability 
modelling  and  estimation  research  pursued  under  contract 
F30602-78-C-0351  during  the  period  October  1978  -  October 
1981.  Two  new  models  of  very  general  applicability  are  intro¬ 
duced  and  the  necessary  mathematical  and  practical  details 
are  developed  in  this  report.  A  new  methodology  for  deter¬ 
mining  vdien  to  stop  testing  and  start  using  software  is  de¬ 
scribed  and  developed.  Finally  a  new  model  for  analyzing  the 
operational  performance  of  a  combined  hardware-software  sys¬ 
tem  is  reported  even  though  it  was  not  a  part  of  the  original 
research  plan. 

2)  TITLE:  COMBINED  HW/SW  RELIABILITY  MODELS 

AUTHOR:  HUGHES  CO.,  L.  JAMES,  J.  BOWEN,  J.  MCDANIEL 

DOC_DATE:  82/4 

SOURCE:  RADC-TR-82-68 

ABSTRACT:  A  general  methodology  is  developed  for  combining  hardware  aund 
software  reliability.  Based  on  this  general  methodology,  a 
baseline  combined  HW/SW  reliability  was  developed  incorpo¬ 
rating  and  unifying  the  SW  reliability  theory  of  Jelinski- 
Moranda,  Geol-Okumoto  with  traditional  HW  reliadjility  theory. 
The  baseline  model  is  computerized  and  includes  various  HW/SW 
failure  and  repair  characteristics,  allowance  for  imperfect 
SW  debugs  auid  modes  of  HW/SW  interaction.  Finally  a  HW/SW 
tradeoff  procedure  is  developed  using  a  combined  HW/SW  avail- 
adsility  measure.  Exaitples  are  provided  to  illustrate  the 
general  theory  and  tradeoff  procedure. 
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3)  TITLE:  EXECUTIVE  SUMMARY  OF  COMMUNICATICWS  PROCESSOR  OPERATING 

SYSTEM  STUDY 

AUTHOR:  JULIAN  GITLIN 

DOC_DATE:  80/11 

SOURCE:  RADC-TR-80-316 

ABSTRACT:  This  report  is  an  executive  summary  prepared  by  the  RADC  pro¬ 

gram  manager  for  the  communications  processing  operating  sys¬ 
tem  program  accomplished  by  Plessey  Fairfield  euid  Data  Indus¬ 
tries  for  RADC  under  contract  F30602-76-C-0456.  The  CPOS 
final  report  consists  of  nine  volumes  which  include  the  major 
technical  areas  of  concern  in  designing  a  secure,  accounta±)le 
and  reliable  operating  system  that  would  control  the  hard¬ 
ware/software  resources  of  an  integrated  switching  node  for 
the  defense  communications  system  in  the  1990' s.  Ttie  CPOS 
final  report  consists  of  nine  volumes: 

1.  Communications  Switch  Operating  System  Study 
Requirements  Analysis 

2.  Software  Relia±)ility  Study 

3.  Security  Considerations  Study 

4.  Operating  System  Survey 

5.  Candidate  Selection 

6.  Implementation  Methods  Study 

7.  Verification  and  Validation 

8.  Design  Specification 

9.  Experimentation 

Although  preparation  of  reliable  software  depends  on  an  un¬ 
derstanding  of  the  type  of  errors  that  occur  euid  their  causes 
fault  analysis  or  failure  mode  analysis  often  are  not  appli¬ 
cable  because  software  errors  are  less  systematic  than  hard¬ 
ware  errors.  The  use  of  a  system-wide  design  procedure  and 
a  program  support  library  have  been  recommended  to  offset 
errors  induced  by  the  individual  skills  and  thought  processes 
of  the  programming  staff.  No  single  reliability  modelling 
approach  has  been  able  to  handle  the  probabilistic  estimates 
of  errors  remaining  in  a  program.  Use  of  a  comprehensive 
error  detection  log  euid  several  models  .T>ervp  as  management 
indicators  of  program  reliability,  but  will  not  serve  as  for¬ 
mal  acceptance  criteria.  Failure  detection  and  recovery  are 
necessary  elements  of  program  design,  but  error  correction  is 
not  recommended.  Implementation  methods  recommended  include 
(1)  imposition  of  DoD  standards  and  guidelines,  (2)  PERT  man¬ 
agement  methodology,  (3)  modular  design,  (4)  top  down  design, 
(5)  HOL,  (6)  structured  prograunming,  (7)  top  down  program¬ 
ming,  (8)  automated  debugging  tools,  and  (9)  formal  design 
methodology. 
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4)  TITLE: 

AUTHOR: 

DOC  DATE: 

SOURCE: 

ABSTRACT: 

5)  TITLE: 


AUTHOR: 


DOC_DATE: 

SOURCE: 

ABSTRACT: 


SSD  SYSTEMS  EFFECTIVENESS  SOFTWARE  RELIABILITY  STANDARD 

LOCKHEED/W,  HANSEN 

80/5/1 

RADC/LOCKHEED  MISSILES  AND  SPACE  COMPANY,  INC. 

This  standard  provides  computer  software  reliability  developn- 
ment  engineering  practices  oriented  toward  modern  development 
techniques  for  use  in  project  planning  and  procurement.  It  is 
designed  to  compliment  the  "SSD  Standards  and  Practices  for 
Software  Engineering"  manual  for  those  projects  with  specific 
quantitative  reli^lbility  requirements  or  goals.  The  overall 
intent  of  this  reliability  standard  is  to  assure  that  relia¬ 
bility  requirements  will  be  achieved  with  a  minimum  of  cost/ 
design/technical  risk.  In  addition,  justifiable  functionally 
equivalent  approaches  or  solutions  will  be  considered.  All 
reliability  procedures  and  techniques  herein  are  intended  to 
allow  equivalent  substitutions  when  approved. 


COMMUNICATIONS  PROCESSOR  OPERATING  SYSTEM  TASK-2  RELIABILITY 
CONSIDERATIONS 

PLESSEY  FAIRFIELD/DATA  INDUSTRIES,  R.  WAXMAN,  R.  DOMITZ, 
GOLDBERG 
80/6 

RADC-TR-80-187,VOL  II  OF  NINE 

This  document,  a  part  of  the  series  whose  executive  summary 
is  covered  in  reference  0003,  attempts  to  define  and  quantify 
the  concept  of  software  reliability.  It  covers  (1)  software 
errors,  (2)  software  reliability  from  the  viewpoint  of  pro- 
grautroing  techniques,  (3)  operational  techniques  for  assuming 
software  with  a  given  level  of  reliability,  and  (5)  integrity 
of  a  system  that  undergoes  catastrophic  failure. 
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6)  TITLE: 

AUTHOR: 

DOC_DATE: 

SOURCE: 

ABSTRACT: 


7)  TITLE: 

AUTHOR: 

DOC_DATE: 

SOURCE: 

ABSTRACT: 


SOFTWARE  RELIABILITY:  REPETITIVE  RUN  EXPERIMENTATIOJ  AND 
MODELING 

BOEING/P.  NAGEL,  J.  SKRIVAN 
82/2 

RADC/NASA  CR-165836 

Boeing  conducted  a  controlled  software-develofwnent  experiment 
in  support  of  software-reliability  estimation  and  modelling. 
Two  programmers  individually  designed  and  coded  three  pro¬ 
grams  each  from  three  specifications.  These  programs  were 
executed  in  repetitive  run  sampling,  where  failure  data  was 
recorded  on  each  of  a  series  of  program  states. 

The  data  was  used  to  verify  that  interfailure  times  are  expo¬ 
nentially  distributed,  to  obtain  estimates  of  the  failure 
rates  of  individual  errors  and  to  demonstrate  how  widely  the 
rates  vary.  This  latter  fact  invalidates  many  of  the  popular 
software  reliability  models  now  in  use.  It  was  observed  that 
the  log  failure  rate  of  interfailure  time  was  nearly  linear 
as  a  function  of  the  number  of  errors  corrected. 

Cox's  proportional  hazards  model  is  proposed  as  a  new  model.. 
Estimates  for  the  unknown  parameters  were  obtained  for  all 
programs.  A  tentative  physical  predictor  was  proposed  based 
on  Halstead's  information  criteria  N  which  might  be  used  in 
forecasting  model  pareimeters. 

PROCEEDINGS  SEMINAR  ON  IMPROVING  AVAILABILITY  OF  HW-SW 
SYSTEMS 

LOS  ANGELES  CHAPTERS  OF  COMPUTER  SOCIETY,  IEEE  ;  RELIABILITY 

SOCIETY 

82/11/13 

RADC 

The  rapid  development  in  Computer  Technology  of  the  past  two 
decades  has  brought  with  it  an  urgent  need  to  provide  ad¬ 
vanced  methods  of  fulfilling  the  potential  of  these  hardware- 
software  systems  in  the  user  environment.  However,  the 
differences  of  skills  euid  working  environments  between  hard¬ 
ware  euid  software  designers  has  frequently  clouded  the  under¬ 
standing,  responsibility  and  needed  contribution  of  each 
discipline's  role. 


ANNOTATED  BIBLIOGRAPHY 


8)  TITLE:  SOFTWARE  RELIABILITY  STUDY 

AUTHOR:  TRW/T.  THAYER 

DOC_DATE:  76/3/19 
SOURCE:  RADC 

ABSTRACT:  A  study  of  software  errors  presented.  Techniques  for  catego¬ 
rizing  errors  according  to  type,  identifying  their  source  and 
detecting  them  are  discussed.  Various  techniques  used  in  an¬ 
alyzing  empirical  error  data  collected  from  four  large  soft¬ 
ware  systems  are  discussed  and  results  of  analysis  are  pre¬ 
sented.  Use  of  results  to  indicate  improvements  in  the  error 
prevention  eind  detection  processes  through  use  of  tool  and 
techniques  is  also  discussed.  A  survey  of  software  relia¬ 
bility  models  is  included,  and  recent  work  on  TRW's  Mathe¬ 
matical  Theory  of  Software  Reliability  (MTSR)  is  presented. 
Finally,  lessons  learned  in  conjunction  with  collecting  soft¬ 
ware  data  are  outlined,  with  recommendations  for  improving 
the  data  collection. 

9)  TITLE:  PERFORMANCE  ENGINEERING  OF  SOFTWARE:  A  CASE  STUDY 

AUTHOR:  C.  SMITH/DUKE  UNIV.  ,  J.  BRCWNE/UNIV.  OF  TEXAS  AT  AUSTIN 

DOC_nATE:  83 

SOURCE:  RADC 

ABSTRACT:  This  paper  summarizes  the  concepts  of  perfornance  engineering 
in  large  software  systems  and  illustrates  the  application  of 
perfornance  engineering  techniques  to  the  early  design  phase 
of  a  large  database  system. 

Performance  engineering  is  a  methodology  for  evaluating  the 
performance  of  software  systems  throughout  their  life  cycles. 
The  case  study  given  here  demonstrates  that  it  is  possible  to 
predict  resource  usage  patterns  of  complex  software  systems 
even  in  early  design  phases  of  the  system,  although  detailed 
predictions  of  resource  usage  are  not  likely  to  be  validated. 
The  results  presented  here  show  the  leverage  of  considering 
performance  implications  in  the  early  design  phases  of  a 
software  project. 
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10)  TITLE: 
AUTHOR: 
DOC_nATE: 
SOURCE: 
ABSTRACT: 


INCREASING  INFORMATION  SYSTEMS  PRODUCTIVITY 

C.  SMITH/DUKE  UNIVERSITY 

81/5 

RADC 

Performance  engineering  is  defined  as  a  practical  discipline 
for  use  through  the  life  cycle  to  increase  information  system 
productivity.  Productivity  is  increased  through  maintenance 
of  output  with  the  same  number  of  personnel,  in  a  shorter 
time,  and  with  a  broader  customer  base  attraction.  Considera¬ 
tions  in  performance  improvement  include:  (1)  Product  use, 

(2)  Design  Definition  and  Alternatives,  (3)  Design  Update 
Definition  and  Alternatives,  (4)  Host  Computer,  and  (5) 
Operational  Constraints.  Ttiese  are  evaluated  in  a  "best 
case"  environment  in  early  stages  of  software  development, 
with  more  detailed  average  performance  analysis  following. 

The  goal  is  to  continually  improve  design.  Factors  critical 
to  success  of  a  performance  engineering  project  are:  (1)  Man¬ 
agement  Commitment,  (2)  Schedule  adjustment,  (3)  Credibility 
of  results,  (4)  Timely  results,  (5)  Justification  of  recom¬ 
mendations,  (6)  Optimistic  vs  realistic  analysis,  and  (7) 
Cooperative  effort.  A  schedule  that  allows  performance  engi¬ 
neering,  based  upon  realistic  use  scenarios,  will  build  a 
higher  quality  product. 


11)  TITLE: 
AUTHOR: 
DOC_DATE: 
SOURCE: 
ABSTRACT: 


SOFTWARE  RELIABILITY  -  BIBLIOGRAPHY 

BALBIR  S.  DHILLON 

01/9 

MMC 

This  paper  presents  a  brief  introduction  and  an  extensive 
bibliography  of  topics  in  software  reliability  and  related 
areas. 


12)  TITLE: 
AUTHOR: 
DOCDATE: 
SOURCE: 
ABSTRACT: 


A  COMPATIBLE  HARDWARE/SOFTWARE  RELIABILITY  PREDICTION  MODEL 

X.  CASTILLO,  T.  SMITH 

81/7/22 

CARNEGI E-MELLON  UNIVERSITY 

A  new  modeling  methodology  to  characterize  failure  processes 
in  Time-Sharing  systems  due  to  hardware  transients  and  soft¬ 
ware  errors  is  presented.  The  basic  assumption  made  is  that 
the  instamtaneous  failure  rate  can  be  approximated  by  a  de¬ 
terministic  function  of  time  plus  a  zero-mean  stationary 
Gaussian  process,  both  depending  on  the  usage  of  the  resource 
considered.  The  probability  density  function  of  the  time  to 
failure  obtained  has  a  decreasing  hazard  function.  Impli¬ 
cations  of  this  methodology  are  discussed. 


ANNOTATED  BIBLIOGRAPHY 


13)  TITLE:  MANAGEMENT  OVERVIEW  OF  SYSTEM  TECHNICAL  SUPPORT  PLAN  FOR 

THE  FIRE-FINDER  SYSTEM  SlJPPORT  CENTER 
AUTHOR:  L.  HESELTC»I/SEMCOR,  INC 

DOC_nATE:  80/8/8 
SOURCE:  NTIS  AI>-A095555 

ABSTRACT:  This  System  Technical  Support  Plein  outlines  the  way  to  cor¬ 
rect  software  problems  on  counter-mortar  euid  counter-artillery 
radars.  It  was  found  that  training  field  repairmen  in  system 
software  was  not  a  cost  effective  way  to  correct  software 
problems.  A  system  support  center  was  established  to  resolve 
software  faults  or  problems  instead.  They  support  field 
personnel  in  determining  if  a  problem  is  caused  by  a  hardware 
or  software  fault,  and  correct  software  faults.  The  su^wrt 
center  determines:  1)  fix,  2)  testing  to  be  performed,  and 
3 )  releases . 

14)  TITLE:  FAULT  DETECTION  EFFICIENCY  MEASUREMENT  VIA  HW  FAULT  SIMULATION 

AUTHOR:  C.  TIMOC/TIMOC  INTERNATIONAL  CO. 

DOC_DATE:  80/3 
SOURCE:  NTIS 

ABSTRACT:  The  overall  objective  of  this  program  is  to  provide  the  means 
for  testing  so  as  to  assure  nearly  fault-free  operation.  A 
more  specific  objective  is  to  measure  the  stuclt  fault  detec¬ 
tion  efficiency  of  the  test  vectors  developed  by  JPL/Hughes 
for  the  MIL-M-38510/470  NASA. 

A  hardware  stuck  fault  simulator  for  the  1802  microprocessor 
was  implemented  and  the  stuck  fault  detection  efficiency  of 
the  test  vectors  developed  by  JPL/Hughes  for  the  MIL-M- 
38510/470  NASA  were  measured  in  three  phases  as  follows: 

Phase  1.  Build  a  breadboard  system  to  perform  the  fault- 
free  function  of  the  1802. 

Phase  2.  Add  fault  simulation  capabilities  to  the  fault- 
free  breadboard. 

Phase  3.  Measure  the  stuck  fault  detection  efficiency  of 
the  test  vectors. 

A  total  of  874  faults  were  injected  into  the  combinatorial 
and  sequential  parts  of  the  RCA  1802  microprocessor  and  it 
was  found  that  39  stuck  faults  were  not  detected.  Therefore, 
the  measured  stuck  fault  detection  efficiency  of  the  MIL-M- 
38510/470  NASA  is  95%.  Since  the  39  undetected  faults  can 
create  catastrophic  errors  in  equipment  designed  for  high 
reliability  applications,  it  is  reconmended  that  the  MIL-M- 
38510/470  NASA  be  enhanced  with  additional  test  vectors  so  as 
to  achieve  100%  stuck  fault  detection  efficiency. 
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TITLE: 

ALTl-HOR: 

DOC^DATE: 

SOURCE: 

ABSTRACT: 


ITTLE: 

AUTHOR: 

OOC_DA'l’E: 

SOURCE: 

ABSTRACT: 


FAULT -TOLERANT  SOFTWARE  FOR  SPACECRAFT  APPLICATIONS 
H.  HECHT/AEROSPACE  CORP. 

75/12/10 

NTIS  AD  A022068 

Fault-tolerant  computers  have  been  developed  for  applications 
that  require  a  very  high  degree  of  hardware  reliability,  and 
it  is  frequently  asked  v^ether  similar  techniques  can  be 
brought  to  bear  on  software  for  critical  applications,  e.g., 
ascent  guidance  software  on  launch  vehicles,  launch-control 
software  for  ground  computers,  and  control  and  comnand  soft¬ 
ware.  The  principal  techniques  employed  in  hardware  fault 
tolerance  are  seen  to  be  applicadjle  also  through  software 
fault  tolerance:  error  detection,  protective  redundancy,  and 
rollback  provisions.  Of  course,  they  need  to  be  implemented 
in  a  specific  manner;  particularly  the  redund^ulcy  must  be 
provided  by  a  different  code  than  that  used  for  the  primary 
modules. 

The  recovery  block  (proposed  by  Randell),  with  the  addition 
of  a  watchdog  timer,  has  been  implemented  in  a  number  of 
skeleton  routines  and  has  been  found  quite  suitable  in  con¬ 
nection  with  the  established  structure  for  spaceborne  soft¬ 
ware. 

A  reliability  model  is  proposed  that  shows  a  very  consider¬ 
able  reduction  in  failure  probability  even  when  the  fault- 
tolerance  provisions  themselves  are  far  from  perfect.  It  is 
therefore  believed  that  the  time  is  quite  ripe  to  undertake 
serious  studies  of  fault- tolerant  software  for  space  appli¬ 
cations. 

THE  COST  OF  SOFTWARE  FAULT  TOLERANCE 

G.  MIGNEAULT/NATIONAL  AERONAUTICS  AND  SPACE  ADMINISTRATION 

82/9 

NTIS 

This  paper  proposed  the  use  of  software  fault  tolerance  tech¬ 
niques  as  a  means  of  reducing  software  costs  in  avionics  and 
as  a  meems  of  addressing  the  issue  of  system  unreliability 
due  to  faults  in  software.  A  model  is  developed  to  provide  a 
view  of  the  relationships  eunong  cost,  redundemcy,  and  relia¬ 
bility  which  suggests  strategies  for  software  development  and 
maintenance  which  are  not  conventional- 

observations  ate  made  about  the  problen  of  escalating  budget 
for  software  auid  about  the  nature  of  some  of  the  causes  of 
increasing  software  cost.  Attention  is  paid  to  schemes  for 
using  dissimilar  redundancy  in  software  to  obtain  a  high 
level  of  reliability. 
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17)  TITLE:  EXCEPTION  HANDLING  AND  SOFTVIARE  FAULT  TOLERANCE 

AUTHOR:  F.  CRISTIAN 

COCCIATE:  82 
SOURCE:  NTIS 

ABSTRACT:  Some  basic  concepts  underlying  the  issue  of  fault  tolerant 

software  design  are  investigated.  Relying  on  these  concepts 
a  unified  point  of  view  on  programmed  exception  handling 
based  on  backward  recovery  is  constructed.  The  cause-effect 
relationship  between  software  design  fault  and  failure  occur¬ 
rences  is  explored  and  a  class  of  faults  for  which  exception 
handling  cein  provide  effective  fault  tolerance  is  charac¬ 
terized.  It  also  shows  that  there  exits  a  second  class  of 
design  faults  which  cannot  be  tolerated  by  using  default  ex¬ 
ception  h£indling.  The  role  that  software  verification  methods 
can  play  in  avoiding  the  production  of  such  faults  is  dis¬ 
cussed. 

18)  TITLE:  PRODUCTION  OF  RELIABLE  FLIGHT  CRUCIAL  SOFTWARE:  VALIDATION 

METHODS  RESEARCH  FOR  FAULT  TOLERANT  AVIONICS  AND  CONTROL 
SYSTEMS  SUB-WORKING  GROUP  MEETING 
AUTHOR:  J.  DUNHAM,  J,  KNIGHT/RESEARCH  TRIANGLE  INST. 

DOC_DATE;  82/5 
SOURCE:  NTIS 

ABSTRACT:  The  state  of  the  art  in  the  production  of  crucial  software 
for  flight  control  applications  was  addressed.  The  associ¬ 
ation  between  reliedsility  metrics  and  software  is  considered. 
Thirteen  software  development  projects  are  discussed.  A  short 
term  need  for  research  in  the  areas  of  tool  development  auid 
software  fault  tolerance  was  indicated.  For  the  long  term, 
research  in  format  verification  or  proof  methods  was  recom¬ 
mended.  Formal  specification  auid  software  reliability  mod¬ 
eling,  were  recommended  as  topics  for  both  short  and  long 
term  research. 
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19)  TITLE: 
AUTHOR: 
DOC_DATE: 
SOURCE: 
ABSTRACT: 


AN  OVERVIEW  OF  RELIABLE  CCMPUTER  SYSTEM  DESIGN 
J.  MC  DONALD/ROYAL  SIGNALS  AND  RADAR  ESTABLISHMENT 
79/11 
NTIS 

This  paper  was  prcxduced  to  support  a  series  of  lectures  on 
reliable  computer  system  design  given  at  a  NATO  ASI  summer 
school  on  multiple  processor  conputers.  The  paper  was  in¬ 
tended  to  be  fairly  self-contained,  but  it  does  lack  a  de¬ 
scription  of  a  practical  fault  tolerant  system. 

The  paper  presents  an  overview  of  reliable  computer  system 
design.  It  attempts  to  provide  a  pragmatic  guide  to  redun¬ 
dancy  and  recovery,  but  does  not  give  a  very  thorough  discus¬ 
sion  of  either  the  theory  or  philosophy  of  relieible  systems. 
The  paper  introduces  and  defines  the  basic  concepts  of  reli¬ 
ability,  and  describes  the  basic  mech2misms  for  achieving 
fault  tolerance.  It  compares  the  attributes  of  multi-pro¬ 
cessor  auid  multi-computer  systems  from  the  point  of  view  of 
reliability.  It  describes  in  some  detail  techniques  for 
achieving  tolera^ice  to  both  hardware  and  software  faults.  The 
paper  concludes  by  outlining  some  of  the  major  unsolved  prob¬ 
lems  of  reliedble  system  design. 


20)  TITLE: 

AUTHOR: 

DOC_DATE: 

SOURCE: 

ABSTRACT: 


ANALYSIS  OF  FAULT  DETECTICX^,  CORRECTION,  AND  PREVENTION  IN 
INDUSTRIAL  COMPUTER  SYSTEMS 
E.  SCHAFFER,  T.  WILLIAMS/PURDUE  UNIV. 

77/9 

NTIS 

This  research  is  concerned  with  three  fault-tolerant  computer 
methods  for  meeting  relieJaility  requirements:  (1)  Hardware 
redundancy  is  defined  as  any  circuitry  in  the  system  not  nec¬ 
essary  for  normal  computer  operation  should  no  faults  occur; 
(2)  Software  redundancy  is  defined  as  additional  progreun 
instructions  present  solely  to  handle  faults;  auid  (3)  Time 
redundancy  is  defined  as  any  retrial  of  instructions.  In 
order  to  provide  an  understanding  of  the  faalt-tolerent  meth¬ 
ods  under  study  today,  examples  of  their  uses  and  limitations 
are  presented.  Hardware  aspects  of  coding  and  modular  redun¬ 
dancy  are  discussed.  Discussions  of  software  include  means  of 
protection,  detection,  and  correction  of  software  faults 
through  software  as  well  software  methods  to  handle  hardware 
errors.  These  methods  include  diagnostics  as  well  as  execu¬ 
tive  recovery  techniques  and  retrials  of  instructions  through 
time  redundancy.  Present  day  computer  capadsilities  also  are 
presented.  Finally,  duplex  &  triplex  fault- tolerant  indus¬ 
trial  computer  systems  are  discussed  that  may  be  built  from 
conventional  computers  with  little  or  no  need  for  expensive 
additional  hardware. 


ANNOTATED  BIBLIOGRAPHY 


21)  TITLE:  EFFECTS  OF  FAILURE  ON  PERFORMANCE  OF  GRACEFULLY  DEGRADABLE 

SYSTEMS 

AUTHOR:  J.  LOSQ/STANFORD  UNIV. 

DOC_DATE:  76/12 
SOURCE:  NTIS  AD-A049849 

ABSTRACT:  The  recent  development  of  multiprocessor  systems  that  offer 
resistance  to  faults  by  gracefully  degrading  after  a  failure 
opens  vast  new  r2uiges  of  applications  for  fault  tolerance  and 
high  reliadDility.  The  paper  presents  a  general  model  for  the 
evaluation  of  such  systems.  It  takes  into  account  the  inter¬ 
nal  structure  of  the  hardware,  the  characteristics  of  the 
various  detection  mechauiisms,  the  unreliability  of  the  soft¬ 
ware  and  even  the  type  of  applications  these  systems  are  used 
for.  It  provides  many  measures  of  the  systems'  performance 
such  as:  availability,  meantime  between  crashes,  average 
processing  power  and  proportion  of  time  spent  in  degraded 
mode.  System  optimization  gives  the  best  values  for  the  num¬ 
ber  of  processors,  memories,  ...,  and  shows  the  trade-offs 
between  hardware  and  software  fault-detection  mechanisms.  The 
model  is  illustrated  by  a  concrete  example. 

22)  TITLE:  SOFTWARE  C3UALITY  ASSURANCE 

AUTHOR:  ED  SOISTMAN 

DOC_DATE:  JUNE  1979 

SOURCE:  UNIVERSITY  OF  CENTRAL  FLORIDA 

ABSTRACT:  The  problems  associated  with  software  development  and  use  are 
investigated  from  a  management  point  of  view.  Having  iden¬ 
tified  the  critical  aspects  of  effective  software  nanagement, 
an  approach  is  suggested  for  the  creation  and  implementation 
of  a  software  quality  assurance  program.  Particular  attention 
is  focused  on  the  concept  of  Life  Cycle  Procurement  as  cur¬ 
rently  utilized  by  the  Department  of  Defense. 

23)  TITLE:  ENGINEERING  RELIABILITY  -  NEW  TECHNIQUES  AND  APPLICATIONS 

AUTHOR:  B.  DHILLON 

DOC_DATE: 

SOURCE:  MMC 

ABSTRACT:  This  article  stresses  that  most  of  the  work  in  the  area  of 

software  reliability  can  be  divided  into  the  following  three 
categories. 

1.  Writing  correct  programs  to  begin  with. 

2.  Testing  the  programs  to  take  out  bugs. 

3.  Modeling  of  software  in  an  attempt  to  predict  its  relia¬ 
bility  and  possibly  study  the  impact  of  related  parameters 

The  following  models  were  discussed  : 

Shoomaun,  Markov  auid  Jelinski-Moranda. 
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THE  SOFTWARE  DEVELOPMENT  NOTEBOOK  -  A  PROVEN  TECHNIQUE 

JOHN  McKISSICK  JR  AND  ROBERT  A.  PRICE 

1979 

PROCEEDINGS  1979  ANNUAL  RELIABILITY  AND  MAINTAINABILITY 
SYMPOSIUM 

The  continuing  need  for  improved  computer  software  demands 
improved  software  developsnent  techniques  such  as  the  Software 
Development  Notebook.  The  organization,  content,  use  and 
audit  of  Software  Development  Notebooks  are  documented  in 
this  paper.  Experience  and  results  from  the  application  of 
this  technique  are  also  presented. 

QUANTITATIVE  SOFTWARE  RELIABILITY  MODELS  -  DATA  PARAMETERS 
DOROTHY  SWEARINGEN  AND  JC«N  DONAHOO 
OCTOBER  1979 

WORKSHOP  ON  QUANTITATIVE  SOFTWARE  MODELS,  NY,  OCT.  1979. 

This  paper  summarizes  the  results  of  a  study  to  identify  data 
requirements  for  software  reliability  modelling.  Brief  de¬ 
scriptions  of  the  models  and  the  data  needed  to  exercize  the 
models  are  presented.  The  paper  concludes  with  a  list  of 
recommendations  for  future  data  collection. 

MODULARITY  IS  NOT  A  MATTER  OF  SIZE 
ROBERT  H.  DUNN  AND  RICHARD  S.  ULLMAN 
1979 

proceedings  1979  ANNUAL  RELIABILITY  AND  MAINTAINABILITY 
SYMPOSIUM 

Division  of  a  conpjter  program  into  a  number  of  smaller  pro¬ 
grams  designated  as  modules  is  a  universally  accepted  prac¬ 
tice  among  software  engineers.  A  modular  architecture  offers 
the  following  advantages: 

-  Parallel  Development 

-  Reduced  program  size  and  costs 

-  Understandability 

-  Reliability 

-  Testing 
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27)  TITLE:  AN  ANALYSIS  OF  ERRORS  AND  THEIR  CAUSES  IN  SYSTEMS  PROGRAMS 

AUTHOR:  ALBERT  ENDRES,  IBM  GERMANY,  BOEBLINGEN,  GERMANY 

DOC_DATE:  APRIL  1975 

SOURCE:  INTERNATIONAL  CONFERENCE  ON  RELIABLE  SOF'IVIARE, APRIL  1975 

ABSTRACT:  Progreuti  errors  detected  during  internal  testing  of  the  oper¬ 
ating  system  DOS/VS  form  the  basis  for  an  investigation  of 
error  distributions  in  system  programs.  Using  a  classifi¬ 
cation  of  errors  according  to  various  attributes,  conclusions 
can  be  drawn  concerning  the  possible  causes  of  these  errors. 
The  information  thus  obtained  is  applied  in  a  discussion  of 
the  most  effective  methods  for  detection  and  prevention  of 
errors. 

28)  TITLE:  METRICS  FOR  EVALl'ATION  OF  QUANTITATIVE  SOFTWARE  MODELLING 

DATABASES 

AUTHOR:  JON  MARTENS,  I IT  RESEARCH  INSTITUTE,  ROME,  NY 

DOC_DATE;  OCTOBER  1979 

SOURCE:  WORKSHOP  ON  QUANTITATIVE  SOFTWARE  MODELS,  NY,  OCT.  1979 

ABSTRACT:  Two  metrics  for  evaluating  software  modelling  databases  are 
developed  in  this  paper.  The  integration  metric  measures  the 
degree  of  data  element  sharing  between  datasets;  the  coverage 
metric  measures  the  datasets'  ability  to  fulfill  a  model's 
requirements. 

29)  TITLE;  SOFTWARE  QUALITY  METRICS  FOR  LIFE-CYCLE  COST-REDUCTION 

AUTHOR:  GENE  F.  WALTERS  AND  JAMES  A.  McCALL 

DOC_DATE:  AUGUST  1979 

SOURCE:  IEEE  TRANSACTIONS  ON  RELIABILITY  VOL  R-28  #3,  AUGUST  1979 

ABSTRACT;  This  paper  identifies  factors  or  characteristics  of  which 
reliaibility  is  one,  which  comprise  the  quality  of  computer 
software.  It  then  discusses  their  impact  over  the  life  of  a 
software  product  and  describes  a  methodology  for  specifying 
them  quantitatively,  including  them  in  system  design,  and 
measuring  them  during  development.  The  methodology  is  still 
experimental,  but  is  rapidly  evolving  toward  application  in 
all  types  of  software.  The  paper  emphasizes  those  factors 
of  software  quality  which  have  the  greatest  importance  at  the 
later  stages  of  a  software  products  life. 
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30)  TITLE:  SPECIAL  SERIES  ON  SYSTEM  INTEGRATION 

AUTHOR:  ELECTRONIC  DESIGN  AND  SYSTEMS  &  SOFTVIARE 

DOC_nATE:  83/4/14 

SOURCE:  MMC 

ABSTRACT:  Tools  have  become  available  to  catch  errors  as  soon  as  pos¬ 
sible  in  the  software  development  cycle.  These  include  the 
familiar  syntax  and  data  type  checkers,  as  well  as  automatic 
change  and  configuration  control  bookkeepers.  Specification 
language  checkers  are  even  available  to  reject  errors  from 
the  start.  The  article  covers  some  of  these  programs,  and 
describes  a  reliability  predictor  that  uses  rate  of  error 
discovery  statistics  to  compute  MTBF.  The  rate  of  closure 
from  the  observed  to  desired  has  been  validated  on  20  pro¬ 
gramming  projects  and  is  described  as  suitable  for  use  in 
scheduling  program  delivery  to  the  customer.  Ada  relia¬ 
bility  aspects  are  also  mentioned. 

31)  TITLE:  A  GENERAL  SOFTWARE  RELIABILITY  MODEL  FOR  PERFORMANCE  PREDICTION 

AUTHOR:  J.  SHANTHI KUMAR/SYRACUSE  UNIV. 

DOC_nATE:  81/3/2 

SOURCE:  MICROELECTRON  RELIABILITY  VOL.  21 

ABSTRACT:  In  this  paper  we  give  a  general  Markov  process  formulation 
for  a  software  reliability  model  and  present  expressions  for 
software  are  performance  measures.  We  discuss  a  general  model 
auid  derive  the  maximum  likelihood  estimates  for  the  required 
parameters  of  this  model.  The  generality  of  this  model  is 
demonstrated  by  showing  that  the  Jelinski-Moranda  model  and 
the  Non-Homogeneous  Poisson  Process  (NHPP)  model  are  both 
very  special  cases  of  our  model.  In  this  process  we  also  cor¬ 
rect  some  errors  in  a  previous  paper  of  the  NHPP  model. 
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32)  TITLE: 
AUTHOR: 
DOC_DATE: 
SOURCE: 
ABSTRACT: 


SOFTViARE  RELIABILITY  -  STATUS  AND  PERSPECTIVES 

C.  RAMAMOORTHY,  F.  BASTANI 

81/12/21 

IEEE  TRANSACTIONS  ON  SOFTVIARE  ENGINEERING  7/82 
It  is  essential  to  assess  the  reliability  of  digital  computer 
systems  used  for  critical  real-time  control  applications 
(e.g,  nuclear  power  plant  safety  control  systems).  This  in¬ 
volves  the  assessment  of  the  design  correctness  of  the  com¬ 
bined  hardware/software  system  as  well  as  the  reliability  of 
the  hardware.  In  this  paper  we  survey  methods  of  determining 
the  design  correctness  of  systems  as  applied  to  computer  pro¬ 
grams.  Automated  program  proving  techniques  are  still  not 
practical  for  realistic  programs.  Manual  proofs  are  lengthy, 
tedious,  and  error-prone.  Software  reliability  provides  a 
measure  of  confidence  in  the  operational  correctness  of  the 
software.  Since  the  early  1970' s  several  software  reliability 
models  have  been  proposed.  We  classify  and  discuss  these 
models  using  the  concepts  of  residual  error  size  and  the 
testing  process  used.  We  also  discuss  methods  of  estimating 
the  correctness  of  the  program  emd  the  adequacy  of  the  set  of 
test  cases  used.  These  methods  are  directly  applicable  to 
assessing  the  design  correctness  of  the  total  integrated 
hardware/software  system  which  ultimately  could  include  large 
complex  distributed  systems. 


33)  TITLE: 

AUTHOR: 

DOC_DATE: 

SOURCE: 

ABSTRACT: 


LIKELIHOOD  FUNCTION  OF  A  DEBUGGING  MODEL  FOR  COMPUTER 
SOFTWARE  RELIABILITY 

B.  LITTLEWOOD,  J.  VERRALL/CITY  UNIVERSITY,  LONDON 
82/6/02 

IEEE  TRANSACTIONS  ON  RELIABILITY  ,  VOL  R-30,  NO.  2,  6/81 
A  simple  model  for  software  reliability  growth,  originally  sug¬ 
gested  by  Jelinski  &  Moranda,  has  been  widely  used  but  suffers 
from  difficulties  associated  with  parameter  estimation.  It  is 
shown  that  a  major  reason  for  obtaining  nonsensical  results  from 
the  model  is  its  application  to  data  sets  which  exhibit  decreas¬ 
ing  reliability.  Presented  is  a  simple,  necessary  and  sufficient 
condition  for  the  maucimum  likelihood  estimates  to  be  finite  and 
suggest  that  this  condition  be  tested  prior  to  using  the  model. 
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34)  TITLE:  A  SUMMARY  OF  THE  DISCUSSION  ON  "AN  ANALYSIS  OF  COMPETING 

SOFTWARE  RELIABILITY  MODELS" 

AUTHOR:  AMRIT  L.  GOEL 

DOC_nATE:  80/9 

SOURCE:  IEEE  TRANSACTIONS  SOFTWARE  ENGINEERING,  9/80 

ABSTRACT:  In  March  1978,  Schick  and  Wolverton  published  a  paper  in  the 
IEEE  Transactions  On  Software  Engineering,  Moranda  (later) 
criticized  several  aspects  of  this  paper.  His  critique  was 
reviewed  by  Littlewood  and  rebutted  by  Schick  and  Wolverton. 
The  purpose  of  this  note  is  to  summarize  and  comment  on  the 
main  points  raised. 

35)  TITLE:  THEORIES  OF  SOFTWARE  RELIABILITY:  HOW  GOOD  ARE  THEY  AND  HOW 

CAN  THEY  BE  IMPROVED 
AUTHOR:  B.  LITTLEWOOD 

DOC_nATE:  79/3/23 

SOURCE:  IEEE  TRANSACTIC^IS  ON  SOFTWARE  ENGINEERING,  9/80 

ABSTRACT:  An  examination  of  the  assumptions  used  in  early  bug-counting 

models  of  software  reliability  shows  them  to  be  deficient. 
Suggestions  are  made  to  improve  modeling  assumptions  and  ex¬ 
amples  are  given  of  mathematical  inplementations.  Model  ver¬ 
ification  via  real-life  data  is  discussed  and  minimum  re¬ 
quirements  are  presented.  An  example  shows  how  these  require¬ 
ments  may  be  satisfied  in  practice.  It  is  suggested  that  cur¬ 
rent  theories  are  only  the  first  step  along  what  threatens  to 
be  a  long  road. 
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A  USER-ORIENTED  SOFTWARE  RELIABILITY  MODEL 

R.  CHEUNG 

79/8/13 

IEEE  TRANSACTIC»JS  OI  SOFTWARE  ENGINEERING 

A  user  oriented  reliability  model  has  been  developed  to  meas¬ 
ure  the  Reliability  that  a  system  provides  to  a  user  com¬ 
munity.  It  has  been  observed  that  in  many  systems,  especially 
software  systems,  reliable  service  can  be  provided  to  a  user 
when  it  is  known  that  errors  exist,  provided  that  the  service 
requested  does  not  utilize  the  defective  parts.  The  relia¬ 
bility  of  service  therefore,  depends  both  on  the  reliability 
of  the  conponents  and  the  probabilistic  distribution  of  the 
utilization  of  the  components  to  provide  the  service.  In 
this  paper,  a  user-oriented  software  reliability  figure  of 
merit  is  defined  to  measure  the  reliability  of  a  software 
system  with  respect  to  a  user  environment.  The  effects  of  the 
user  profile,  vdiich  summarizes  the  characteristics  of  the 
users  of  a  system,  on  system  reliability  are  discussed.  A 
simple  Markov  model  is  formulated  to  determine  the  relia¬ 
bility  of  a  software  system  based  on  the  reliability  of  each 
individual  module  and  the  measured  intermodular  transition 
probabilities  as  the  user  profile.  Sensitivity  analysis  tech¬ 
niques  are  developed  to  determine  modules  most  critical  to 
system  reliability.  The  applications  of  this  model  to  develop 
cost-effective  testing  strategies  and  to  determine  the  ex¬ 
pected  penalty  cost  of  failures  are  also  discussed.  Some 
future  refinements  eund  extensions  of  the  model  are  presented. 

MODELS  FOR  HARDWARE-SOFTWARE  SYSTEM  OPERATIONAL-PERFORMANCE 
EVALUATION 

A.  GOEL,  J.  SOENJOTO/SYRACUSE  UNIVERSITY 
81/5/18 

IEEE  TRANSACTIONS  ON  RELIABILITY,  8/81 

Stochastic  models  for  hardware-software  systems  are  developed 
and  used  to  study  their  performance  as  a  function  of  hard¬ 
ware-software  failure  and  maintenance  rates.  Expressions  are 
derived  for  the  distribution  of  time  to  a  specified  number  of 
software  errors,  system  occupancy  probabilities,  system  reli¬ 
ability,  availability,  and  average  availeibility.  The  behavior 
of  these  measures  is  investigated  via  numerical  exanples. 
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STOCHASTIC  RELIABILITY-GROWTH:  A  MODEL  FOR  FAULT-REMOVAL  IN 
COMPUTER  PROGRAMS  AND  HARDWARE-DESIGNS 
B.  LITTLEWOOD/CITY  UNIVERSITY,  LONDON 
81/1/12 

IEEE  TRANSACTIONS  ON  RELIABILITY,  10/81 

An  assumption  commonly  made  in  early  models  of  software  reli¬ 
ability  is  that  the  failure  rate  of  a  program  is  a  constant 
multiple  of  the  (unknown)  number  of  faults  remaining.  This 
inplies  that  all  faults  contribute  the  same  amount  to  the 
failure  rate  of  the  program.  The  assunption  is  challenged 
and  an  alternative  proposed.  The  suggested  model  results  in 
earlier  fault-fixes  having  a  greater  effect  them  later  ones 
(the  faults  which  make  the  greatest  contribution  to  the  over¬ 
all  failure  rate  tend  to  show  themselves  earlier,  and  so  are 
fixed  earlier),  and  the  DFR  property  between  fault  fixes 
(assurance  about  programs  increases  during  periods  of  fail¬ 
ure-free  operation,  as  well  as  at  fault  fixes).  The  model 
is  tractable  auid  allows  a  total  execution  time  to  achieve  a 
target  reliability,  and  total  number  of  fault  fixes  to  target 
reliability,  are  obtained.  The  model  might  also  apply  to 
hardware  reliability  growth  resulting  from  the  elimination  of 
design  errors. 

AN  ERROR  DETECTION  MODEL  FOR  APPLICATIC»I  DURING  SOFTWARE 
DEVELOPMENT 

P.  MORANDA/McDONALD  DOUGLAS  ASTRONAUTICS  COMPANY 
81/1/29 

IEEE  TRANSACTIONS  W  RELIABILITY,  10/81 

A  variation  of  the  Jelinski/Moranda  model  is  described.  The 
main  feature  of  this  new  model  is  that  the  expanding  size  of 
a  developing  program  is  accommodated,  so  that  the  quality  of 
a  program  can  be  estimated  by  analyzing  an  initial  segment  of 
the  written  code.  Two  parameters  are  estimated.  The  data 
are:  a)  time  separations  between  error  detections,  b)  the 
number  of  errors  per  written  instruction,  c)  the  failure  rate 
(or  finding  rate)  of  a  single  error,  euid  d)  a  time  record  of 
the  number  of  instructions  under  test.  This  model  permits 
predictions  of  MTTF  euid  error  content  of  software. 
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40)  TITLE: 

RELIABILITY  AND  MAINTAINABILITY  OF  ELECTRONINC  SYSTEMS 

tv 

CHAPTER  6  SOFTWARE  RELIABILITY  AND  MAINTAINABILITY 

AUTHOR: 

J.  ARSENAULT 

DOC  DATE: 

79/1 

SOURCE: 

A.  FLOWERS 

ABSTRACT:  This  article  expounds  on  attributes  which  may  he  employed  in 
the  generation  of  reliable  and  maintainable  software  in¬ 
cluding:  modularity,  emphasis  on  system  design  not  coding, 
top  down  approach  and  structured  programming. 


41)  TITLE:  IEEE  RELIABILITY  SOCIETY  NEWSLETTER 

AUTHOR:  lEEE/SUSAN  EAMES,  EDITOR 

DOC_DATE:  83/4 

SOURCE:  MMC-E.  GRIFFIN 

ABSTRACT:  This  doojunent  consists  of  chapter  meeting  notes  and  a  status 
summary  for  1982. 


42)  TITLE: 
AUTHOR; 


DOCJDATE: 

SOURCE: 

ABSTRACT: 


HARDWARE/SOFTWARE  FMECA 

FRED  M.  HALL;  EVALUATION  RESEARCH  CORP.;  ARLINGTON 
RAYMCWD  A.  PAUL;  SURFACE  WEAPONS  CENTER;  DAHLGREN 
WENDY  E.  SNOW;  EVALUATION  RESEARCH  CORP.;  ARLINGTON 
JANUARY  1983 

1983  PROCEEDINGS  ANNUAL  RELIABILITY  AND  MAINTAINABILITY 
SYMPOSIUM 

(AUTHORS)  This  paper  describes  procedures  which  can  be  used  to 
determine  reliability  REQUIREMENTS  for  both  hardware  and  soft¬ 
ware  elements  of  a  system  which  incorporates  an  embedded  com¬ 
puter  subsystem.  The  analysis  technique,  Hardware/Software 
Failure  Modes,  Effects  and  Criticality  Analysis  (FMECA)  may  be 
utilized  to  ENSURE  reliable  software  throughout  the  software 
developmenmt  cycle;  including  requirements  definition,  coding, 
checkout  and  software  test  . . .  H/S  FMECA  provides  a  method  for 
examining  software  errors  at  the  highest  functional  levels,  then 
progressively  tracking  errors  into  lower  level  functions  . 


43)  TITLE: 

AUTHOR: 

DOC_DATE: 

SOURCE: 

ABSTRACT: 


GROUND  LAUNCHED  ASSAULT  BREAKER  FLIGHT  PROGRAM  PROBLEM  REPORT 
FILE 

E.L.  GRIFFIN 
AUGUST  1982 

VARIOUS  MARTIN  MARIETTA  DEVELOPMENTAL  TEAM  PERSONNEL 
This  log  contains  the  complete  set  of  166  software  problem 
reports  written  against  the  14K  Ground  Launched  Assault 
Breaker  Missile  Flight  Program.  The  problem  reports  reflect 
hardware,  software,  and  coiifcination  hardware/software  prob¬ 
lems  for  a  very  complex  real-time  program  that  navigates  to 
a  target  based  on  airborne  target  update  data  and  dispenses 
subinunitions  on  a  tank. 
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44)  TITLE:  CAN  SOFTWARE  BENEFIT  FROM  HARDWARE  EXPERIENCE 

AUTHOR:  HERBERT  HECHT 

DOC_DATE:  1975 

SOURCE:  PROCEEDINGS  1975  ANNUAL  RELIABILITY  AND  MAINTAINABILITY 

SYMPOSIUM 

ABSTRACT:  (AUTHOR)  The  question  posed  in  the  title  is  euiswered  with  a 
qualified  "yes".  While  software  requires  a  completely  dif¬ 
ferent  attack  on  failure  mechanisms  than  used  for  hardware, 
there  is  considerable  commonality  in  reliability  organization 
and  procedures,  and  some  in  the  systems  reliability  area.  The 
present  lack  of  a  generally  accepted  metric  for  software  re¬ 
liability  inpedes  the  information  transfer  from  the  hardware 
field.  Steps  towards  a  quantifitative  measure  for  software 
reliability  are  outlines. 

45)  TITLE;  CW  SOFTWARE  COMPLEXITY 

AUTHOR;  L.  A.  BELADY 

DOC_DATE;  OCTOBER  1979 

SOURCE:  WORKSHOP  ON  QUANTITATIVE  SOFTWARE  MODELS,  N.Y.,  OCT.  1979 

ABSTRACT:  In  1974  there  was  very  little  known  work  on  complexity  in 

general  and  even  less  on  the  complexity  of  programming.  At 
that  time  Professor  Beilner  and  myself  shared  an  office  at 
Imperial  College  and  according  to  our  mutual  interest,  dis¬ 
cussed  problems  of  "conplex"  programs  and  of  large  scale  pro¬ 
gramming.  ..  .We  found  a  sizeable  body  of  work  on  computational 
complexity  (and  after  5  year's  work  (ecs))  ...  we  learned 
that  conplexity  can  be  perceived  in  at  least  two  different 
ways:  sometimes  it  appears  as  a  measure  of  uncertainty  or 
surprise  and  sometimes  it  is  deterministic  and  is  defined  as 
a  count  or  magnitude.  ...  The  objective  of  this  paper  is  to 
give  a  brief  and  a  little  bit  organized  survey  of  this 
"conplex"  repertoire  of  approaches. 

46)  TITLE:  IN  SEARCH  OF  SOFTWARE  COMPLEXITY 

AUTHOR:  DR.  BILL  CURTIS,  GENERAL  ELECTRIC,  ARLINGTON  VA 

DOC_DATE;  OCTOBER  1979 

SOURCE:  WORKSHOP  ON  QUANTITATIVE  SOFTWARE  MODELS  N.Y. ,  OCT.  1979 

ABSTRACT:  This  paper  is  a  summary  of  a  tutorial  given  on  the  state-of- 
the-art  in  software  conplexity  research...  Different  types  of 
conplexity  metrics  are  reviewed  along  with  a  framework  for 
studying  large  compilations  of  metrics.  Ultimately,  a  defi¬ 
nition  is  posed  on  interactional  terms  which  encompasses  di¬ 
verse  areas  within  the  field. 
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47)  TITLE:  AN  ANTI -COMPLEXITY  EXPERIMENT 

AUTHOR:  L.  A.  BELADY 

DOC_nATE:  OCTOBER  1979 

SOURCE:  WORKSHOP  ON  QUANTITATIVE  SOFTWARE  MODELS,  N.Y.,  OCT.  1979 

ABSTRACT:  _ [I]n  the  early  70's  systematic  studies  of  large  programs 

uncovered  an  established  the  evolutionary  nature  of  software, 
namely  that  programs  are  not  static  objects  but  undergo  con¬ 
tinuous  modification  to  cope  with  the  everchanging  environ¬ 
ment.  ..  [Cjonplexity,  a  term  v^ich  in  this  context  informally 
describes  the  difficulty  of  predictably  manipulating  soft¬ 
ware. 

48)  TITLE:  SOFTWARE  FAULT  TOLERANCE:  WHY  IT'S  NECESSARY  AND  A 

METHODOLOGY 

AUTHOR:  MYRON  HECHT,  SOHAR  INCORPORATED,  LA  JOLLA,  CA 

DOC_nATE:  13  NOV  1982 

SOURCE:  PROCEEDINGS,  SEMINAR  ON  IMPROVING  AVAILABILITY  OF  HARDWARE- 

SOFTWARE  SYSTEMS,  IEEE  COMPUTER  SOCIETY,  LOS  ANGELES 
ABSTRACT:  As  real  time  control  requirements  grow  more  complex  and  pro¬ 
gressively  more  demands  are  placed  on  the  computer  system, 
software  related  failures  will  become  increasingly  important 
contributors  to  the  total  number  of  system  failures.  This 
trend  is  already  evident  in  the  error  data  collected  from 
several  highly  reliable  conputer  systems.  Unfortunately,  the 
conplete  prevention  of  faults  in  complex  software  is  beyond 
present  technical  capabilities.  Under  these  circumsteinces, 
critical  applications  must  provide  for  the  toleration  of 
software  faults  (in  addition  to  hardware  faults).  The  recov¬ 
ery  block  concept  presents  a  widely  applicable  means  for 
implementing  fault  tolerance  within  a  program. 
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49)  TITLE: 
AUTHOR: 
DOC_nATE: 
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ABSTRACT: 


CARE  III  FINAL  REPORT 

L.  BRYANT,  Bryant,  L.  GUCCIONE,  J.  STIFFLER 
79/11 

NTIS/N80-15423 

This  report  describes  the  work  done  during  the  first  phase  of 
a  two-phase  effort  to  develop  a  computer  program  to  aid  in 
assessing  the  reliability  of  fault- tolerant  avionics  systems. 
The  overall  effort  consists  of  five  major  tasks:  1)  Este±)lish 
the  basic  requirements  that  must  be  satisfied  if  the  program 
is  to  achieve  its  overall  objective.  2)  Define  a  general 
program  structure  consistent  with  these  requirements.  3) 
Develop  and  program  a  mathematical  model  relating  the  relia¬ 
bility  of  a  fault-tolerant  system  to  the  (not  necessarily 
time-independent)  failure  rates  and  coverage  factors  charac¬ 
terizing  its  various  elements.  4)  Develop  and  program  a 
mathematical  model  for  evaluating  the  coverage  (probability 
of  successful  recovery)  associated  with  any  given  fault  as  a 
function  of  the  type  and  location  of  the  fault,  the  appli¬ 
cable  fault  detection  and  isolation  mechanism,  and  the  number 
and  status  of  prior  faults.  5)  Develop  and  program  a  pro¬ 
cedure  whereby  a  user  of  these  models  cam  accurately  euid 
conveniently  specify  the  configuration  of  the  system  to  be 
evaluated  and  the  constraints  influencing  its  ability  to 
recover  from  faults. 

The  first  three  of  these  tasks  were  completed  during  Phase 
One;  the  resulting  requirements,  program  structure,  and  reli¬ 
ability  model  are  discussed  in  detail  in  Volume  I  of  this 
report,  along  with  the  tradeoffs  and  sample  reliability  as¬ 
sessments  made  in  arriving  at  the  approach  finally  taken.  The 
Computer  Program  Requirements  Document  is  contained  in  Volume 
II.  This  latter  volume  also  includes  several  appendices  con¬ 
taining  computer  print-outs  euid  other  ancillary  material  sup¬ 
porting  the  conclusions  presented  in  Volume  I. 
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50)  TITLE:  FACTORS  IN  SOFTOARE  QUALITY 

AUTHOR:  JIM  A.  McCALL,  PAUL  K.  RICHARDS,  GENE  F.  WALTERS 

DOC_DATE:  NOVEMBER  1977 

SOURCE:  RADC  TR-77-369,  VOLUMES  I, II  &  III 

ABSTRACT:  An  hierarchial  definition  of  factors  affecting  software  qual¬ 
ity  was  compiled  after  an  extensive  literature  search.  The 
definition  covers  the  complete  range  of  software  development 
and  is  broken  down  into  non-oriented  and  software-oriented 
characteristics.  For  the  lowest  level  of  the  software-ori¬ 
ented  factors,  metrics  were  developed  that  would  be  indepen¬ 
dent  of  the  programming  language.  These  measurable  criteria 
were  collected  and  validated  using  actual  Air  Force  data 
bases.  A  haindbook  was  generated  that  will  be  useful  to  the 
Air  Force  meinagers  for  specifying  overall  quality  of  the 
software  system. 

51)  TITLE;  SYSTEM  HARDWARE  AND  SOFTWARE  RELIABILITY  ANALYSIS 

AUTHOR:  WILLIAM  E.  THOMPSON;  COLUMBIA  RESEARCH  CORPORATICW; ARLINGTON 

DOC_DATE;  1983 

SOURCE;  1983  IEEE  PROCEEDINGS  OF  THE  ANNUAL  RELIABILITY  AND 
MAINTAINABILITY  SYMPOSIUM 

ABSTRACT:  This  paper  presents  a  practical  model  and  procedure  for  ana¬ 
lyzing  combined  hardware  and  software  reliability  of  embedded 
computer  systems.  The  emphasis  is  on  combined  hardware  and 
software  relie±)ility  qualification  testing. 

52)  TITLE:  MIL-S-XXXXX:  SOFTWARE  RELIABILITY  AND  .MAINTAINABILITY 

SPECIFICATION 

AUTHOR:  ELECTRONIC  SYSTEMS  DIVISICW,  HANSCOM  FIELD,  MASS 

DOC_DATE:  OCTOBER  1983 

SOURCE :  RADC/RBET 

ABSTRACT:  This  specification  requires  the  estciblishment  eind  implemen¬ 
tation  of  a  Software  Reliability  and  Maintainability  (R&M) 
Program  by  the  contractor.  The  purpose  of  this  program  is  to 
assure  that  software  developed,  acquired  or  otherwise  pro¬ 
vided  under  this  contract  complies  with  the  requirements  of 
the  contract.  It  is  intended  that  the  program  be  effectively 
tailored  and  economically  planned  and  developed  in  consonance 
with,  or  eui  extension  of  the  contractor's  other  reliability, 
maintainability,  quality  assurance,  administrative  and  tech¬ 
nical  programs.  When  referenced, - this  specification  shall 

apply  to  the  acquisition  of  software  either  acquired  alone  or 
as  part  of  a  system.  (It).. shall  also  apply  to  all  deliver¬ 
able  design,  test,  maintenance  and  support  software  developed 
under  this  contract.  For  purposes  of  this  specification,  the 
term  software  includes  firmware. 
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53)  TITLE:  SOFTWARE  RELIABILITY  ESTIMATION:  A  REALIZATION  OF 

COMPETING  RISK 
AUTHOR:  WAY  KUO 

DOC_nATE:  SPRING  1983 

SOURCE:  MICROELECTRONICS  AND  RELIABILITY  JOURNAL 

ABSTRACT:  A  software  reliability  model  presented  here  assumes  a  time- 
dependent  failure  rate  and  that  debugging  can  remove  as  well 
as  add  faults  with  a  non-zero  probability.  This  paper  pro¬ 
poses  a  con^und-distribution  software  reliability  model.  We 
make  three  assumptions, 

1)  The  software  originally  contains  X(0)  bugs  where  X(0) 
is  a  constant  to  be  determined, 

2)  Each  time  interval  between  failures  has  a  nonconstant 
occurrence  rate, 

3)  When  a  software  fault  is  fixed,  an  additional  error  can 
be  introduced,  the  probability  of  correcting  or  intro¬ 
ducing  errors  follows  a  discrete  distribution, 

54)  TITLE:  ESTIMATING  SOFTWARE  DEVELOPMENT 

AUTHOR:  E,  B,  DALY 

DOC_nATE:  JULY  1979 

SOURCE:  IEEE 

ABSTRACT:  Programming  effort  can  be  estimated  and  evaluated  in  terms  of 

instructions  generated  per  hour.  Although  this  methodology 
of  judging  software  effort  has  been  shown  to  be  very  effective 
it  must  be  utilized  with  great  care  since  rates  vary  dramati¬ 
cally  among  software  jobs  having  different  design  objectives, 
different  program  complexity  emd  differennt  resources, 

55)  TITLE:  STATISTICAL  PREDICTIC^I  OF  PROGRAMMING  ERRORS 

AUTHOR:  R,  W,  MOTLEY  AND  W,  D.  BROOKS 

DOC_DATE:  30  NOV  1976 

SOURCE:  RADC-TR-77-175 

ABSTRACT:  The  need  for  developing  new  tools  and  techniques  for  pro¬ 
ducing  more  reliable  low  cost  software, ,, has  led  to  attempts 
to  analyze  the  nature  and  types  of  software  errors  in  order 
to  be  able  to  accurately  predict  the  reliability  of  the  soft¬ 
ware  product _ The  report  focuses  on  the  einalysis,  using 

multiple  linear  regression  techniques,  of  software  error  data 
and  related  structural,  complexity,  and  programmer-related 
varied3les  extracted  from  two  large  DoD  coimmand  and  control 
software  pojects  totalling  over  250,000  lines  of  higher  order 
language  source  code,  Ttiis  eight  month  study  focused  on  the 
statistical  prediction  of  programming  errors  using  a  wide 
range  of  progreim  structure/complexity  variables  and  selected 
progreunmer  variables  as  predictors. 
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56)  TITLE: 

AUTHOR: 

DOC_DATE: 
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ABSTRACT: 


DESIGN  AND  CODE  INSPECTIONS  TO  REDUCE  ERRORS  IN  PROGRAM 
DEVELOPMENT 
M.  E.  FAGAN 
MARCH  1976 

REPRINT  FROM  IBM  SYSTEMS  JOURNAL,  VOL.  15,  NO.  3,  1976 
Substantial  net  improvements  in  programming  quality  and  pro¬ 
ductivity  have  been  obtained  through  the  use  of  formal  in¬ 
spections  of  design  and  of  code.  Improvements  are  made  pos¬ 
sible  by  a  systematic  and  efficient  design  and  code  verifica¬ 
tion  process,  with  well-defined  roles  for  inspection  partic¬ 
ipants.  The  mauiner  in  vrtiich  inspection  data  is  categorized 
and  made  suitable  for  process  emalysis  is  an  important  factor 
in  attaining  the  improvements.  It  is  shown  that  by  using 
inspection  results,  a  mechanism  for  initial  error  reduction 
followed  by  ever-inproving  error  rates  can  be  achieved. 


57)  TITLE: 

AUTHOR: 

DOC_DATE: 

SOURCE: 

ABSTRACT: 


MIL-STD-1644(TD) ,  TRAINER  SYSTEM  SOFTOARE  ENGINEERING 
RE5QUIREMENTS 

NAVAL  TRAINING  EQUIPMENT  CENTER,  ORLANDO,  FL  32813 
15  JAN  1982 

Ibe  purpose  of  this  standard  is  to  establish  uniform  require¬ 
ments  for  the  development  and  documentation  of  trainer  system 
software.  These  requirements  shall  apply  to  trainer  system 
software  (including  firmware  and  microcode)  which  is  devel¬ 
oped  either  alone  or  as  a  portion  of  a  trainer  system  or  sub¬ 
system  development.  Only  unmodified  computer  vendor  commer¬ 
cial  software  will  be  exempt  from  the  development  and  docu¬ 
mentation  requirements  of  this  stemdard. 
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ABSTRACT: 


SOFTWARE  CONSIDERATIONS  IN  AIRBORNE  SYSTEMS  AND 
EQUIPMENT  CERTIFICATION 
SPECIAL  COMMITTEE  OF  RTCA 
NOVEMBER  1981 

RADIO  TECHNICAL  COMMISSION  FOR  AERONAUTICS  (RTCA) 
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61)  TITLE: 
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THE  MEASUREMENT  AND  MANAGEMENT  OF  SOFTVIARE  RELIABILITY 
JOHN  D.  MUSA 
SEPTEMBER  1980 
PROCEEDINGS  OF  THE  IEEE 

The  theme  of  this  paper  is  the  field  of  software  reliability 
measurement  eind  its  applications.  Needs  for  and  potential 
uses  of  software  reliability  measurement  are  discussed.  Soft¬ 
ware  reliability  and  hardware  reliability  are  compared,  eind 
some  basic  software  reliability  concepts  are  outlined.  A 
brief  summary  of  the  major  steps  in  the  history  and  evolution 
of  the  field  is  presented.  IVo  of  the  leading  software  reli¬ 
ability  models  are  described  in  some  detail.  The  topics  of 
combinations  of  software  (and  hardware)  components  and  avail- 
a±)ility  are  discussed  briefly.  The  paper  concludes  with  am 
analysis  of  the  current  state  of  the  art  and  a  description  of 
further  research  needs. 

VALIDITY  OF  EXECUTION-TIME  THEORY  OF  SOFTVIARE  RELIABILITY 
JOHN  D.  MUSA 
AUGUST  1979 

IEEE  TRANSACTIONS  ON  RELIABILITY 

This  paper  investigates  the  validity  of  the  execution-time 
theory  of  software  reliability.  The  theory  is  outlined,  along 
with  appropriate  bac)cground,  definitions,  assumptions,  and 
mathematical  relationships.  Both  the  execution  time  euid  cal¬ 
endar  time  component  are  described.  The  inportant  assunptions 
are  discussed.  Actual  data  are  used  to  test  the  validity  of 
most  of  the  assumptions.  Model  and  actual  behavior  are  com¬ 
pared.  The  development  projects  and  operational  confutation 
center  software  from  which  the  data  have  been  obtained  are 
characterized  to  give  the  reader  some  basis  for  judging  the 
breadth  of  applicability  of  the  concepts. 

OPERATIWS  RESEARCH  AND  RELIABILITY 
DANIEL  GROUCHKO  (EDITOR) 

JUNE/JULY  1969 

PROCEEDINGS  OF  A  NATO  CONFERENCE 
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62)  TITLE:  RELIABILITY  MODEL  DEMONSTRATION  STUDY,  VOLUMES  I  &  II 

AUTHOR:  J.  E.  ANGUS,  J.  B.  BCWEN,  S.  J.  VANDENBERG( HUGHES  AIRCRAFT) 

DOC_DATE:  AUGUST  1983 

SOURCE:  ROME  AIR  DEVELOPMENT  CENTER 

ABSTRACT:  This  report  contains  the  results  of  a  study  to  determine  the 
use  and  applicability  to  Air  Force  software  acquisition  man¬ 
agers  of  six  quantitative  software  reliability  models  to  a 
major  command,  control,  communications,  eind  intelligence  CCCI 
system.  The  scope  of  the  study  included  the  collection  of 
software  error  data  from  an  ongoing  ccci  project,  fitting 
six  software  reliability  models  to  the  data,  analyzing  the 
predictions  provided  by  the  models,  and  developing  conclu¬ 
sions,  recommendations,  and  guidelines  for  software  acquisi¬ 
tion  managers  pertaining  to  the  use  euid  applicability  of  the 
models. 

63)  TITLE:  COMMENTS  ON  HIGHLY  RELIABLE  SOFTOARE  FOR  AVIONICS 

APPLICATIONS 

AUTHOR:  JACOB  T.  SCHWARTZ 

DOC_DATE:  SEPTEMBER  23,  1981 

SOURCE:  INSTITUTE  FOR  COMPUTER  APPLICATICW  IN  SCIENCE  AND  ENGINEERING 

ABSTRACT:  The  differences  between  hardware  and  software  reliability  for 
digital  systems  are  discussed  in  the  context  of  applications 
where  a  failure  may  result  in  the  loss  of  human  life.  In  par¬ 
ticular,  it  is  argued  that  techniques  for  guareinteeing  relia¬ 
bility  for  hardware  are  not  necessarily  appropriate  for  soft¬ 
ware.  The  potential  of  a  variety  of  approaches  for  assuring 
software  reliability  is  discussed. 
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64)  TITLE:  FITTING  AN  EXPONENTIAL  SOFTWARE  MODEL  TO  FIELD  FAILURE  DATA 

AUTHOR:  R.  W.  SCHMIDT 

DOC_DATE:  MAY  1982 

SOURCE:  OFFICE  OF  NAVAL  RESEARCH 

ABSTRACT:  The  quantitative  prediction  and  measurement  of  software  reli¬ 
ability  is  of  vital  importance  in  the  development  of  high 
quality  cost  effective  software.  Many  software  reliability 
models  have  been  postulated  in  the  literature,  however  few 
have  been  applied  to  field  data.  A  model  based  upon  the  as¬ 
sumption  that  the  failure  rate  of  the  software  is  propor¬ 
tional  to  the  number  of  residual  software  errors  leads  to  a 
constant  failure  rate  and  an  exponential  reliability  func¬ 
tion.  The  model  contains  two  constants:  the  proportionality 
constant  K  and  the  initial  (total)  number  of  errors  Et. 

The  constants  K  and  Et  can  be  estimated  during  early  design 
by  comparison  of  the  present  project  with  historical  data. 
During  the  integration  test  phase,  a  jnore  accurate  determi¬ 
nation  of  the  model  parameters  can  be  obtained  by  using  sim¬ 
ulator  test  data  as  if  it  were  operational  failure  data.  The 
simulator  data  is  collected  at  two  different  points  in  the 
integration  test  phase  and  the  two  parameters  can  be  deter¬ 
mined  from  moment  estimator  formulas.  The  more  powerful  max¬ 
imum  likelihood  method  can  also  be  employed  to  obtain  point 
and  interval  estimates.  It  is  also  possible  to  use  least 
squares  methods  to  obtain  parameter  estimates  which  is  the 
simplest  method  and  provides  insight  into  the  analysis  of  the 
data. 

This  report  utilizes  a  set  of  software  development  and  field 
data  taken  by  John  D.  Musa  as  a  vehicle  to  study  the  ease  of 
calculation  and  the  correspondence  of  the  three  methods  of 
parameter  estimation.  The  sensitivity  of  the  reliability  pre¬ 
dictions  to  parameter  changes  are  studied  and  compared  with 
field  results. 

The  results  show  if  data  is  carefully  collected,  software 
reliability  models  are  practical  and  yield  useful  results. 
These  can  serve  as  one  measure  to  help  in  choosing  among 
competitive  designs  and  as  a  gauge  of  when  to  terminate  the 
integration  test  phase. 

65)  TITLE:  IEEE  COMPUTER  SOCIETY  SOFTOARE  RELIABILITY  MEASUREMFImT 

WORKING  GROUP 

AUTHOR: 

D0C_DATE;  AUGUST  1982 

SOURCE: 

ABSTRACT: 
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SOFTWARE  ACQUISITION  MANAGEMENT  GUIDEBOOK:  SOFTWARE  QUALITY 
ASSURANCE 

GEORGE  NEIL  AND  HARVEY  I.  GOLD 
AUGUST  1977 

ELECTRONICS  SYSTEMS  DIVISION 

This  report  is  one  of  a  series  of  Software  Acquisition  Man¬ 
agement  Guidebooks  which  provide  information  and  guidance  for 
ESD  Program  Office  personnel  who  are  charged  with  planning 
and  managing  the  acquisition  of  command,  control,  and  commu¬ 
nications  system  software  procured  under  Air  Force  800  series 
regulations  and  related  software  acquisition  management  con¬ 
cepts.  It  provides  guidance  for  establishing  and  imple¬ 
menting  a  software  quality  assurance  program  which  it  dis¬ 
cusses  in  terms  of  Program  Office  quality  assurance  require¬ 
ments  (as  defined  by  AFR  74-1  eund  ESDM  74-1),  contractor 
quality  assurance  requirements  as  defined  by  MIL-S-52779(AD) , 
eind  software  quality  assurance  at  ESD.  Special  attention  is 
given  to:  (1)  the  relationship  of  quality  assurance  to  other 
acquisition  management  disciplines;  (2)  the  integration  of 
quality  assurance  requirements  into  the  system;  (4)  monitor¬ 
ing  the  implementation  of  quality  assurance  requirements;  emd 
(5)  common  problems  and  proposed  solutions. 
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JOINT  LOGISTICS  COMMANDERS  (JLC) 

MAJ.  LARRY  A.  FRY;  AIR  FORCE;  ANDREWS  AFB 
1981 

1981  PROCEEDINGS  ANNUAL  RELIABILITY  AND  MAINTAINABILITY 
SYMPOSIUM 

In  /^ril  1979,  the  Computer  Software  Management  (CSM)  Siib- 
group,  working  under  the  auspices  of  the  Joint  Logistics 
Commanders  (JLC)  Joint  Policy  Coordinating  Group  for  Computer 
Resource  Memagement  (JPCG-CRM),  conducted  a  workshop  to  re¬ 
view  current  DOD  policy,  procedures  and  standards  in  the  area 
of  software  mauiagement.  Panels  were  organized  to  review  the 
areas  of  software  acquisition  and  development  standards, 
software  documentation,  standards  for  software  quality  and 
software  accepteuice  criteria.  The  findings  and  recommen¬ 
dations  from  this  workshop  are  summarized  in  this  paper. 
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EVALUATING  AUTOMATABLE  MEASURES  OF  SOFTWARE  DEVELOPMENT 

VICTOR  R.  BASILI  AND  ROBERT  W.  REITER,  JR 

1979 

IEEE 

There  is  a  need  for  distinguishing  a  set  of  useful  automat- 
eJale  measures  of  the  software  development  process  and  pro¬ 
duct.  Measures  are  considered  useful  if  they  are  sensitive  to 
externally  observable  differences  in  development  environments 
euid  their  relative  values  correspond  to  some  intuition  re¬ 
garding  these  characteristic  differences.  Such  measures  could 
provide  an  objective  quantitative  foundation  for  constructing 
quality  assurance  standards  and  for  calibrating  mathematical 
models  of  software  reliability  eind  resource  estimation.  This 
paper  presents  a  set  of  automateOsle  measures  that  were  imple¬ 
mented,  evaluated  in  a  controlled  experiment,  and  found  to 
satisfy  these  usefulness  criteria.  The  measures  include  com¬ 
puter  job  steps,  program  changes,  program  size,  and  cyclo- 
matic  conplexity. 


A  CRITIQUE  OF  THE  JELINSKI-MORANDA  MODEL  FOR  SOFTWARE 
RELIABILITY 

BEV  LITTLEWOOD,  THE  GEORGE  WASHINGTON  UNIVERSITY;  WASHINGTON 

D.C.  AND  THE  CITY  UNIVERSITY;  LONDON 

1901 

1981  PROCEEDINGS  ANNUAL  RELIABILITY  AND  MAINTAINABILITY 
SYMPOSIUM 

The  paper  discusses  some  problems  associated  with  an  early 
model  for  software  reliability  growth  during  debugging,  first 
proposed  by  Jelinski  and  Moranda.  It  is  suggested  that  the 
assumption  of  all  faults  contributing  equally  to  the  overall 
failure  rate  is  overly  naive  and  can  be  improved.  Necessary 
and  sufficient  conditions  are  given  for  ML  estimates  of  the 
model  parameters  to  be  finite.  It  is  shown  that  there  is 
always  a  finite  prob2djility  that  N  -  .  Since  other  authors 

have  shown  from  simulated  data  that  ML  estimates  can  be  mis¬ 
leading  even  when  finite,  it  is  important  that  goodness-of- 
fit  tests  are  performed.  Such  a  test  on  one  set  of  data 
shows  the  model  to  perform  badly;  an  alternative  model  due  to 
Littlewood  and  Verrall  is  better. 
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70)  TITLE:  THE  MANY  FACETS  OF  QUANTITATIVE  ASSESSMENT  OF  SOFTWARE 

RELIABILITY 

AUTHOR:  J.-C.  RAULT,  IRIA  -  INSTITUT  DE  RECHERCHE  d ' INFORMATIQUE  ET 

d'AUTOMATIQUE  DOMAINE  DE  VOLUCEAU  ,  FRANCE 
DOC_DATE:  1979 
SOURCE:  IEEE 

ABSTRACT:  After  recalling  the  prevailing  techniques  and  procedures  for 
attaining  reliable  and  high  quality  software  (qualitative, 
preventive,  fault-avoidance,  fault-tolerance) ,  approaches  to 
quauititative  assessing  software  reliedaility  are  considered. 
Two  main  types  of  approach  are  distinguished  and  surveyed: 
extension  of  hardware-based  techniques  (application  of  con¬ 
ventional  reliability  theory,  extension  of  hardware  test  ef¬ 
ficiency  measurement  to  software)  and  techniques  specific  to 
software  (correlating  structural,  textual  and  behavioral  com¬ 
plexity  measures  to  some  measure  of  reliedsility) .  Scope  and 
domains  of  application  of  these  approaches  are  delineated. 
Then,  a  comprehensive  scheme  for  assessing  software  relia¬ 
bility  that  attenpts  to  reconcile  the  various  reliability 
models  surveyed  is  proposed.  As  a  conclusion,  one  discusses 
a  possible  incorporation  of  the  proposed  scheme  into  current 
programming  stations  with  a  view  to  extending  their  capeJail- 
ities  in  both  software  project  management  and  software  qual¬ 
ity  assurance. 

71)  TITLE:  TESTING  FOR  SOFTWARE  RELIABILITY 

AUTHOR:  J.  R.  BROWN  AND  M.  LIPCW;  TRW  SYSTEMS 

DOC_DATE; 

SOURCE: 

ABSTRACT:  This  paper  presents  a  formulation  of  a  novel  methodology  for 
evaluation  of  testing  in  support  of  operational  reliability 
assessment  and  prediction.  The  methodology  features  an  incre¬ 
mental  evaluation  of  the  representativeness  of  a  set  of  de¬ 
velopment  and  validation  test  cases  together  with  definition 
of  additional  test  cases  to  enheuice  those  qualities.  Several 
techniques  vhich  permit  specification  of  expected  operational 
usage  are  described.  An  experimental  application  of  the 
techniques  to  a  small  program  is  provided  as  an  illustration 
of  the  proposed  use  of  the  methodology  for  operational  soft¬ 
ware  reliedjility  estimation. 
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DESIOJ  OF  SELF-CHECKING  SOFTWARE 
S.  S.  YAU  AND  R.  C.  CHEUNG 

PROCEEDINGS  OF  INTERNATIOlAL  CONFERENCE  CW  RELIABLE  SOFTWARE 
This  paper  discussed  different  techniques  for  constructing  a 
piece  of  self-checking  software  for  systems  where  ultra-reli- 
edjility  is  required.  Self-checking  software  can  be  designed 
to  detect  software  errors,  to  locate  and  to  stop  the  propa¬ 
gation  of  software  errors  and  to  verify  the  integrity  of  the 
system. 

FAULT-TOLERANT  SOFTWARE 

HERBERT  HECHT,  SENIOR  MEMBER  IEEE;  SOHAR  INC.,  LOS  ANGELES 
AUGUST  1979 

IEEE  TRANSACTIONS  ON  RELIABILITY 

Limitations  in  the  current  capabilities  for  verifying  pro¬ 
grams  by  formal  proof  or  by  exhaustive  testing  have  led  to 
the  investigation  of  fault-tole ranee  techniques  for  applica¬ 
tions  vdiere  the  consequence  of  failure  is  particularly  se¬ 
vere.  Two  current  approaches,  N-version  programming  and  the 
recovery  block,  are  described.  A  critical  feature  in  the 
latter  is  the  acceptance  test,  and  a  number  of  useful  tech¬ 
niques  for  constructing  these  are  presen^d.  A  system  reli¬ 
ability  model  for  the  recovery  block  is  introduced,  euid  con¬ 
clusion  derived  from  this  model  that  affect  the  design  of 
fault-tole rant  software  are  discussed. 

ORGANIZING  FOR  SUCCESSFUL  SOFTWARE  DEVELOPMENT 
EDMUND  B.  DALY 
DECEMBER  1979 
DATAMATION 

Software  development  requires  competent  technologists,  compe¬ 
tent  managers  and  an  effective  organization  structure.  A  good 
organization  structure  is  meaningless  without  a  well-defined 
design  methodology  and  without  effective  management  prac¬ 
tices.  The  organization  structure  brings  together  technol¬ 
ogists  and  management,  but  the  structure  must  work  within  the 
culture  of  the  orgawiization. 
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75)  TITLE:  MODIFIED  MUSA  THEORETIC  SOFIWARE  RELIABILITY 

AUTHOR:  H.  B.  CHENOWETH,  PH.D.;  WESTINGHOUSE  ELECTRIC  CORP. 

DOC_DATE:  1981 

SOURCE:  1981  PROCEEDINGS  ANNUAL  RELIABILITY  AND  MAINTAINABILITY 

SYMPOSIUM 

ABSTRACT:  Presented  in  this  paper  is  an  analysis  of  the  underlying  as¬ 
sumptions  and  constraints  relative  to  the  Musa  software  exe¬ 
cution  time  reliability  model.  The  theory  was  examined  to 
validate  the  ruiderlying  basic  assumptions  with  the  view  of 
adding  greater  generality.  The  resulting  modifications  in  the 
model  theoretic  analysis  technique  and  parameter  definition 
that  result  from  a  particular  subclass  of  programs  are  ex¬ 
plained  in  terms  of  their  contribution  to  the  predicted  MTTF 
and  the  expected  time  to  complete  testing. 

76)  TITLE:  REPORT  ON  THE  DELIBERATIONS  OF  THE  SOFTWARE  RELIABILITY 

WORKING  GROUP 

AUTMOR:  M.  LIPOW,  CHAIRMAN;  TRW  SYSTEMS  AND  ENERGY  GROUP,  REDONDO 

BEACH,  CA 

DOC_DATE:  1979 

SOURCE:  IEEE 

ABSTRACT:  Recommendations  that  software  and  hardware  reliability  be 

considered  in  a  systems  context  were  presented  to  the  Wor)ring 
Group.  Similarities  and  differences  of  hardware  and  software 
characteristics  as  elements  of  reliability  models  and  as  they 
shape  terminology  were  suggested  as  the  main  topics  to  be 
discussed. 

The  Wor)<ing  Group's  primary  recommendations  were  that  a  new 
IEEE  Tas)c  Group  be  established  to  formulate  and  support  soft¬ 
ware  reliability  data  collection  methodology,  that  the  IEEE 
Terminology  Task  Group  be  supported,  and  that  several  ex¬ 
isting  software  reliability  mcxlels  are  usable,  and  should  be 
applied. 

77)  TITLE:  A  SOFTWARE  EVALUATION:  RESULTS  AND  RECOMMENDATIONS 

AtrrHOR:  JAN  M.  HOWELL;  U.S.  AIR  FORCE;  EDWARDS  AFB 

DOCDATE:  1983 

SOURCE:  1983  PROCEEDINGS  ANNUAL  RELIABILITY  AND  MAINTAINABILITY 

SYMPOSIUM 

ABSTRACT:  This  paper  discusses  a  software  quality  evaluation  conducted 
on  a  major  U.S.  Air  Force  avionics  system.  The  originally 
planned  effort  was  limited  to  a  hardware  evaluation  but  was 
expanded  to  include  software  when  the  impact  of  software  be¬ 
came  apparent.  The  paper  presents  some  results  obtained, 
discusses  pattern  software  problems  encountered  and  suggests 
ways  to  avoid  those  repetitive  difficulties. 
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IMPLEMENTATION  AND  MEASURABLE  OUTPUT  OF  SOFTWARE  QUALITY 
ASSURANCE 

P.  CASTIGLIONE,  W.  THOMPSON;  GENERAL  ELECTRIC  CO. , BINGHAMTON 
1983 

1983  PROCEEDINGS  ANNUAL  RELIABILITY  AND  MAINTAINABILITY 
SYMPOSIUM 

This  paper  describes  the  Software  Quality  Assurance  (SQA) 
controls  and  activities  for  all  elements  of  the  software  de¬ 
sign,  development,  and  manufacturing  process  used  at  General 
Electric  Aerospace  Control  Systems  Department. 

WORKING  GROUP  ON  SOFTWARE  COST 

C.  E.  WALSTCW,  IBM,  FEDERAL  SYSTEMS  DIVISION,  BETHESDA,  MD 

1979 

IEEE 

This  paper  summarizes  and  highlights  the  discussions  that 
took  place  in  the  working  group  on  software  cost.  A  number  of 
issues  were  identified  relating  to  the  state  of  the  art  in 
software  cost  estimation  and  to  its  implications  for  devel¬ 
oping  and  using  software  cost  models. 

SOFTWARE  SAFETY:  A  DEFINITION  AND  SOME  PRELIMINARY  THOUGHTS 

NANCY  G.  LEVESON;  DEPT.  OF  INFORMATION  AND  COMPUTER  SCIENCE 

UNIVERSITY  OF  CALIFORNIA;  IRVINE,  CA  92717 

APRIL  1981 

HUGHES  AIRCRAFT  CO. 

Software  safety  is  the  subject  of  a  research  project  in  its 
initial  stages  at  the  University  of  California  Irvine.  This 
research  deals  with  critical  real-time  software  where  the 
cost  of  an  error  is  high,  e.g.  human  life.  In  this  paper 
software  techniques  having  a  bearing  on  safety  are  described 
amd  evaluated.  Initial  definitions  of  software  safety  con¬ 
cepts  are  presented  along  with  some  preliminary  thoughts  and 
research  questions. 
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81)  TITLE:  THE  CURRENT  STATE  OF  SOFTWARE  RELIABILITY  MODELING 

AUTHOR:  V.  VERMURI;  DEPT.  OF  COMPUTER  SCIENCE,  STATE  UNIVERSITY  OF  NY 

BINGHAMTON,  NY  13901 
DOC_nATE:  1979 
SOURCE:  IEEE 

ABSTRACT:  The  develojxnent  of  reliable  software  is  a  complex  process. 

This  complexity  stems  from  its  large  size,  strong  human  ele¬ 
ment  in  its  design  and  development  and  the  loncertainties  as¬ 
sociated  with  its  operational  environment.  The  problem  of 
building  normative  models  to  characterize  software  develop¬ 
ment  process,  in  order  to  predict  its  reliaJaility,  cost, 
utility,  etc.,  is  therefore  an  inherently  complex  process. 

The  difficulties  are  further  exacerbated  by  the  lack  of  stan¬ 
dardized  practices.  Furthermore,  the  remge  of  validity  of 
most  of  the  error  frequency  models  is  confined  to  the  imme¬ 
diate  environment  within  which  they  were  developed.  Their 
sensitivity  to  parameters  and  test  data  remains  to  be  tested. 
It  appears,  therefore,  that  the  time  is  ripe  for  the  develop¬ 
ment  of  an  integrated  approach  which  brings  together  the  con¬ 
cepts  such  as  conplexity,  reliability,  utility,  and  cost. 
Within  such  a  framework,  it  is  possible  to  find  a  proper 
niche  for  the  current  crop  of  operational,  develojanental  and 
maintenance  models.  Tools  and  techniques  for  such  an  inte¬ 
grated  approach  are  available  in  the  body  of  knowledge  var¬ 
iously  known  as  general  systems  theory  or  the  theory  of  mod¬ 
eling  of  complex  systems. 
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EVOLUTION  OF  QUALITY/  RELIABILITY  DUE  TO  LITIGATION 
RICHARD  M.  JACOBS,  PE,  CONSULTANT  SERVICES  INSTITUTE,  INC., 
LIVINGSTON;  JOHN  MIHALSKY,  ED.  D. ,  NEW  JERSEY  INSTITUTE  OF 
TECHNOLOGY,  NEWARK 
1983 

1983  PROCEEDINGS  ANNUAL  RELIABILITY  AND  MAINTAINABILITY 
SYMPOSIUM 

During  the  past  twelve  to  fifteen  years,  manufacturers  eind 
sellers  of  products  have  been  bombarded  with  litigation  in¬ 
volving  their  product.  During  this  period  some  management 
groups  have  recognized  that  it  taltes  twenty  to  twenty-five 
times  more  in  sales  to  pay  for  the  cost  of  a  litigation  or 
claim.  This  does  not  count  the  roughly  $10,000  of  internal 
costs  that  companies  have  been  experiencing  to  service  their 
own  litigation  staff. 

Included  in  the  internal  costs  of  retrieving  information  are 
the  following  items: 

failure  reports,  inspection  records,  tests  records, 
drawings,  specifications,  operating  procedures,  orgeui- 
izational  charts,  patents,  every  engineering  change  issued 
for  the  product,  every  manufacturing  change,  all  purchase 
orders,  receiving  tickets,  stockroom  records,  service 
records  for  the  item  involved,  internal  memoraunda,  engi¬ 
neering  note  books,  log  books,  calibration  records,  per¬ 
sonnel  records  (for  those  who  are  associated  with  the 
item  involved). 

The  Quality  Control  and  Reliability  Specialist  in  every  phase 
of  the  operation  is  dirctly  involved  in  the  recording  of  the 
data  contained  in  the  documents  listed  edxjve  auid  in  some  re¬ 
spect  becomes  associated  with  the  accuracy  eind  the  repeat- 
etbility  of  these  data.  In  addition  to  the  litigation  re¬ 
trieval  costs,  there  are  now  laws  passed  by  various  Federal 
and  State  Legislatures  to  which  the  companies  must  comply. 
Evidence  of  this  conpliance  most  frequently  originates  or 
passes  through  the  hands  of  the  Quality  and  Reliability 
Specialist. 
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ABSTRACT:  This  paper  will  address  the  pitfalls  associated  with  estab¬ 
lishing  and  managing  a  Conpjter  Software  Assurance  Program 
(CSAP).  The  paper  will  be  based  on  the  experience  gained 
with  the  CSAP  Program  on  the  General  Dynamics  F-16  Multi¬ 
national  Staged  Inprovement  Program  (MSIP).  The  basic  ques¬ 
tion  addressed  is  how  a  CSAP  can  be  structured  so  that  the 
pitfalls  can  be  minimized  or  avoided  and  how  the  productivity 
can  be  iirproved  by  directing  the  CSAP  effort  to  not  only  sat¬ 
isfy  the  requirements  of  MIL-S-52779A  but  at  the  saune  time  to 
help  (rather  tham  hinder)  the  software  developnent  effort. 

84)  TITLE:  PROGRAM  CONTROL  COMPLEXITY  AND  PRODUCTIVITY 

AUTHOR;  J,  E.  GAFFNEY,  JR,  IBM,  FEDERAL  SYSTEMS  DIVISION,  MANASSAS, 
VIRGINIA  22110 
DOC_nATE:  1979 
SOURCE:  IEEE 

ABSTRACT;  This  paper  describes  two  sinple  measures  of  Program  Control 
Complexity  and  indicates  their  relationship  to  Programmer 
Productivity.  This  work  is  based  upon  a  limited  amount  of 
real  program  data  euid  is  related  to  the  earlier  work  of 
McCabe  and  Chen. 


85)  TITLE:  AN  INDEX  OF  COMPLEXITY  FOR  STRUCTURED  PROGRAMS 

AUTHOR:  IRENE  L.  STORM  AND  STANLEY  PREISER,  UNIVAC  AND  POLYTECHNIC 

INSTITUTE  OF  NEW  YORK 
DOC_DATE:  1979 
SOURCE:  IEEE 

ABSTRACT:  An  index  of  complexity  for  structured  programs  is  introduced. 

It  provides  an  "a  prior"  measure  of  program  complexity,  thus 
signaling  the  programmer  when  che  program  "prob^ly"  exceeds 
the  limit  of  "easy"  comprehension.  In  general,  the  index  of 
complexity  for  structured  programs  is  less  than  the  index  of 
conplexity  for  unstructured  programs. 
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SWEDISH  HARDWARE/SOFTWARE  RELIABILITY 

LEIF  E<RISTIANSEN;  ERICSSON  DEFENSE  AND  SPACE  SYSTEMS;  MOELNDAL 
1983 

1983  PROCEEDINGS  ANNUAL  RELIABILITY  AND  MAINTAINABILITY 
SYMPOSIUM 

This  paper  presents  the  application  of  reliability  in  a  de¬ 
fense  industry.  Section  1  describes  significant  events  in  the 
reliability  area  at  our  company.  Section  2  presents  Relia¬ 
bility  Prediction,  Maintainability  Analysis  and  Reliability 
Growth  used  by  our  company.  The  last  section  describes  a 
method  for  objective  control  of  software  production. 
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HARDWARE/SOFTWARE  AVAILABILITY  FOR  A  PHONE  SYSTEM 

RICHARD  PISKAR;  ISKRA  TELEMATIKA;  KRANJ 

1983 
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The  paper  predicts  the  availability  of  a  digital  telephone 
decentralized  system  composed  of  computer  controlled  switch¬ 
ing  modules  and  duplicated  intermodule  PCM  switches.  There 
are  known  hardware  failure  rates  of  modules  and  PCM  switches 
euid  estimated  software  failure  rates.  The  system  is  repair¬ 
able  with  different  repair  rates  for  modules  and  PCM 
switches.  The  failure  and  repair  rates  are  constant,  3u:id  the 
distributions  of  times  to  faults  are  negative  exponential. 

The  problem  is  to  determine  the  availability  of  the  system. 
Markov  process  model  with  discrete  states  and  continuous  time 
is  used  to  determine  hardware  and  software  availability.  In 
steady  state  the  probabilities  of  each  state  are  derived  from 
a  system  of  simulteuneous  equations.  Combining  the  probedail- 
ities  of  the  states  with  the  number  of  subscribers  or  lines 
affected,  the  analytical  expressions  for  hardware  euid  soft¬ 
ware  availabilities  of  the  system  are  derived.  The  analytical 
expression  of  general  decentralized  and  distributed  system 
availability  emd  average  times  to  fault  in  dedicated  parts  of 
the  system  are  the  results.  The  hardware/software  model  is 
offering  the  einalytical  relation  between  reliability  and  pro¬ 
ductivity,  or  creation  of  economic  value  during  production 
and  use  of  the  system. 
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WILLIAM  E.  THOMPSON,  COLUMBIA  RESEARCH  CORP.,  ARLINGTON 
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ABSTRACT:  This  paper  presents  a  Bayesian  availability  model  for  combined 
hardware  and  software  systems.  Such  systems  are  sometimes 
called  embedded  con^ter  systems.  The  system  model  presented 
assumes  that  each  embedded  computer  system  malfunction  can  be 
related  to  one  of  three  sources:  (1)  hardware,  (2)  software, 
or  (3)  an  unknown  source.  The  procedures  presented  in  this 
paper  can  also  serve  as  the  basis  for  system  specifications, 
warranty  provisions,  or  other  contractual  agreements  related 
to  combined  hardware  and  software  system  availeibility. 
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89)  TITLE:  HARDWARE-SOFTWARE  AVAILABILITY:  A  COST  BASED  TRADE-OFF  STUDY 

AUTHOR;  AMRIT  L.  GOEL  ,  SYRACUSE  UNIVERSITY,  SYRACUSE; 

JOPIE  B.  SOENJOTO,  CENTRAL  BUREAU  OF  STATISTICS,  JAKARTA 
DOC_nATE:  1983 

SOURCE:  1983  PROCEEDINGS  ANNUAL  RELIABILITY  AND  MAINTAINABILITY 

SYMPOSIUM 

ABSTRACT:  Software  has  become  a  major  source  of  system  malfunctions  and 
a  prime  contributor  to  the  overall  cost  of  maintaining  large 
commerical  and  weapons  systems.  This  paper  addresses  the  pro¬ 
blem  of  assessing  the  reliability  emd  availability  of  such 
systems.  A  cost  model  is  developed  to  study  the  trade-off 
between  the  hardware  and  software  subsystems  for  cases  where 
such  trade-offs  are  permissible.  The  basic  approach  follows 
the  model  developed  by  Goel  eund  Soenjoto 
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RELIABILITY  OF  SHUTTLE  MISSION  CONTROL  CENTER  SOFTVJARE 
MARTIN  L.  SHOOMAN,  POLYTECHNIC  INSTITUTE  OF  NEW  YORK , BROOKLYN 
GEORGE  RICHESON,  NATA,  HOUSTON 
1983 
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This  paper  presents  the  results  of  a  study  made  of  the  relia¬ 
bility  of  software  for  the  Space  Shuttle  Mission  Control 
Center  Data  Processing  Complex.  The  ground  based  software, 
which  is  approximately  1.2  million  lines  of  source  code,  was 
used  to  simulate  the  mission  prior  to  flight,  for  use  in 
flight  controller  and  astronaut  training.  During  the  course 
of  the  simulation,  all  discrepancies  from  correct  behavior 
were  reported,  and  subsequently  diagnosed  as  due  to  hardware, 
software,  or  operator  errors.  The  model  predictions  are  com¬ 
pared  with  the  performance  of  this  software  during  the  first 
shuttle  flight. 
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A  GUIDEBOOK  FOR  SOFTWARE  RELIABILITY  ASSESSMENT 
AMRIT  L.  GOEL,  SYRACUSE  UNIVERSITY 
AUGUST  1983 

ROME  AIR  DEVELOPMENT  CENTER,  GRIFFISS  AFB,  NY 
The  purpose  of  this  guideboo)c  is  to  provide  state-of-the-art 
information  about  the  selection  and  use  of  existing  software 
reliability  models.  Towards  this  objective,  we  have  presented 
a  brief  summary  of  the  available  models  backed  by  a  detailed 
discussion  of  most  of  the  models  in  the  appendices. 

One  of  the  difficulties  in  choosing  a  model  is  to  find  a 
match  between  the  testing  environment  and  a  class  of  models. 
To  help  a  user  in  this  process,  we  have  presented  a  detailed 
discussion  of  most  of  the  assumptions  that  characterize  the 
various  software  reli^±>ility  models.  The  process  of  devel¬ 
oping  a  model  has  been  explained  in  detail  and  illustrated 
via  numerical  exanples. 
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AUTHOR:  G.  BENYCW-TINKER,  DEPT.  OF  COMPUTING  AND  CONTROL,  IMPERIAL 

COLLEGE,  LCMXDN 

DOC_DATE: 

SOURCE: 

ABSTRACT:  One  large  (and  fionctionally  complicated)  program  has  been 

studied  in  an  attempt  to  find  a  measure  of  complexity  which 
reflects  the  effort  required  to  understand  it.  Examination  of 
the  program  showed  that  a  major  part  of  the  effort  was  re¬ 
lated  to  the  way  in  which  the  component  procedures  interacted 
functionally  via  the  calling  hierarchy.  This  suggested  a  new 
way  to  measure  the  complexity.  Each  of  the  13  released  ver¬ 
sions  of  the  program  was  analysed  to  determine  the  structure 
of  the  procedure  calling  hierarchy.  The  evolution  of  the 
complexity  measure  obtained  indicated  a  modest  increase  in 
complexity  of  understanding  which  was  consistent  with  other 
evidence.  Procedures  also  interacted  strongly  through  shar¬ 
ing  access  to  global  variables,  emd  a  new  approach  to  clas¬ 
sifying  the  complexity  of  such  interactions  is  proposed. 
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DORMANCY  AND  POWER  ON-OFF  CYCLING  EFFECTS  ON  ELECTRONIC 
EQUIPMENT  AND  PART  RELIABILITY 

J.  A.  BAUER,  D.  F.  COTTRELL,  T.  R.  GAGNIER,  E.‘  W.  KIMBALL, 
et  al,  MARTIN  MARIETTA  AEROSPACE,  ORLANDO,  FL 
AUGUST  1973 

ROME  AIR  DEVELOPMENT  CENTER,  GRIFFISS  AFB,  NY 
Martin  Marietta  Aerospace  has  conducted  two  12-montii  pro¬ 
grams.  The  first  was  to  collect,  study,  and  analyze  relia¬ 
bility  information  and  data  on  dormant  military  electronic 
equipment  and  parts  and  to  develop  current  dormant  failure 
rates,  factors,  and  prediction  techniques.  The  second  was 
to  collect,  study,  and  analyze  reliability  information  and 
data  on  military  electronic  systems  subjected  to  power  on-off 
cycling,  to  correlate  failure  incidence  with  power  on-off 
cycling,  and  to  quantify  power  on-off  cycling  affects  with 
respect  to  the  dormancy  and  operating  states. 
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ASSESSMENT  OF  SOFTOARE  RELIABILITY 

G.  J.  SCHICK  AND  R.  W.  WOLVERIW,  TRWSYSTEMS  ENGINEERING  AND 
INTEGRATION  DIVISION,  LOS  ANGELES  AND  REDONDO  BEACH 
SEPTEMBER  1972 

This  paper  discusses  nvethods  for  and  problems  in  achieving 
reliability  of  large-scale  software  systems.  Comparative 
studies  were  made  of  a  U.S.  Air  Force  software  project,  a 
NASA  software  project,  and  a  commercial  software  project. 
Software  developnent  and  test  management  procedures  which 
lead  to  software  reliability  are  analyzed.  The  underlying 
premise  is  that  software  relicibility  must  be  designed  into 
the  system  from  the  outset  using  a  systems  approach.  The 
systems  approach  to  achieving  software  reliability  requires 
(1)  understanding  of  the  total  software  development  euid  test 
life  cycle,  (2)  identification  of  conventional  and  extended 
conventional  test  techniques  for  precision  validation  testing 
of  applications  programs,  and  (3)  allocation  of  resources  in 
a  cost-performance-effective  roeuiner,  in  adveuice,  over  the 
entire  development  period. 
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A  DETERMINISTIC  MODEL  TO  PREDICT  "ERROR- FREE"  STATUS  OF  COMPLEX 
SOFTWARE  DEVELOPMENT 

IRWIN  NATHAN,  SR  MEMBER  IEEE,  XEROX  CORPORATIC»4 

1979 

IEEE 

A  top)-down  model  for  evaluating  software  "relicibility"  is 
propxDsed  and  tested  against  seven  case  histories.  The  model 
is  shown  to  accurately  predict  when  the  software  cein  be  ex 
pected  to  be  reasonably  "error-free".  The  model  is  based 
upon  the  work  of  19th  century  actuary  by  the  name  of  Benjamin 
Gomp>ertz.  The  model  was  then  used  in  a  forecasting  mode  for 
an  on-going  software  evaluation. 
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SOPIVIARE  RELIABILITY 

MARTIN  L.  SHOOMAN,  POLYTECHNIC  INSTITUTE  OF  NEW  YORK; 
MYRON  LIPOW,  'rRW,INC.,  REDCMX)  BEACH,  CA 


A.  Basics 

.  Terminology 

.  Hardware  -  Software  comparisons 
.  Reliability  as  a  quality  attribute 

B.  Control  of  Software  Errors 
.  Sources  of  errors 

.  Prevention  and  detection  of  errors 

C.  Reliability  Models  and  Measurements 
.  Complexity  models 

.  Empirical,  structural,  and  statistical  models 

D.  Applications  and  Case  Histories 

E.  Questions  and  Discussion 

A  WORKABLE  SOFIWARE  QUALITY/RELIABILITY  PLAN 
ROBERT  H, ,  DUNN  AND  RICHARD  S.  ULLMAN 
1978 

1978  PROCEEDINGS  ANNUAL  RELIABILITY  AND  MAINTAINABILITY 
SYMPOSIUM 

The  twin  problems  of  the  reliability  emd  maintainability  of 
embedded  software  are  viewed  as  susceptible  to  solution  by 
the  disciplines  of  built-in  quality  assurance.  Several  facets 
to  such  an  approach  are  presented.  Emphasis  is  placed  on  the 
use  of  appropriate  techniques  and  tools,  with  audits  seen  as 
the  principal  device  for  assuring  a  well-planned  and  orderly 
executed  development  cycle.  The  thrust  of  the  paper  is  to¬ 
ward  pragmatic  solutions.  Thus,  the  array  of  techniques  and 
tools  described  is  restricted  to  those  that  are  readily  a- 
vailable  and  have  previously  met  with  success  in  the  develop¬ 
ment  of  embedded  software. 
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QUANTITATIVE  SOFTWARE  COMPLEXITY  MODELS:  A  PANEL  SUMMARY 

VICTOR  R.  BASILI,  DEPT  OF  COMPUTER  SCIENCE,  UNIVERSITY  OF 

MARYLAND 

1979 

IEEE 

Several  participants  at  the  conference  formed  a  panel  on 
software  complexity  measures.  The  following  topics  were 
discussed: 

.  Defining  A  Software  Complexity  Measure 
.  Developing  A  Software  Measure 
.  Using  A  Software  Measure 
.  The  Effect  of  Software  Metrics 

AIRBORNE  SYSTEMS  SOFTWARE  ACQUISITICW  ENGINEERING  GUIDEBOOK 
FOR  QUALITY  ASSURANCE. 

M.  LIPCW,  TRW  DEFENSE  AND  SPACE  SYSTEMS  GROUP 
NOVEMBER  1977 

AEROIAUTICAL  SYSTEMS  DIVISION,  WRIGHT  PATTERSON  AFB,  OHIO 
This  report  is  one  of  a  series  of  guidebooks  v^ich  provide 
guidance  for  ASD  and  SAMSO  Program  Office  and  engineering 
personnel  in  the  acquisition  management  and  engineering  of 
Airborne  Systems  software  procured  under  Air  Force  800  series 
regulations.  It  provides  information  that  will  help  personnel 
plan,  specify,  and  monitor  quality  assurance  activities  in 
connection  with  the  acquisition  of  Computer  Program  Configu¬ 
ration  Items  (CPCI's)  for  Airborne  Systems. 

ELEMENTS  OF  SOFTWARE  SCIENCE 

MAURICE  H.  HALSTEAD 

1977 

This  book  contains  the  first  systematic  summarization  of  a 
branch  of  experimental  auid  theoretical  science  dealing  with 
the  human  preparation  of  computer  programs  and  other  types  of 
written  material.  Application  of  the  classical  methods  of  the 
natural  sciences  demonstrates  that  even  such  relatively  in¬ 
tangible  objects  as  written  abstracts  and  computer  programs 
are  governed  by  natural  laws,  both  in  their  preparation  and 
in  their  ultimate  form. 

The  work  underlying  each  chapter  of  this  monograph  is  firmly 
based  on  the  methods  and  principles  of  classical  experimental 
science.  EVen  so,  the  results  in  this  area,  or  more  specif¬ 
ically,  the  concept  that  significant  quantitative  results 
are  attainable  in  such  an  area,  are  sufficiently  coionter- 
intuitive  as  to  appear  almost  weird. 
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101)  TITLE:  WHEN  AND  HCW  TO  USE  A  SOFTWARE  RELIABILITY  MODEL 

AUTHOR:  AMRIT  L,  GOEL,  VICTOR  R.  BASILI,  AND  PETER  M.  VALDES 

DOC_DATE:  DECEMBER  1982 

SOURCE:  GODDARD  SPACE  FLIGHT  CENTER 

ABSTRACT':  Meuny  analytical  models  were  proposed  during  the  last  decade 
for  software  reliability  assessment.  These  models  served  a 
useful  purpose  in  identifying  the  need  for  an  objective  ap¬ 
proach  to  determining  the  quality  of  a  software  system  as  it 
goes  through  various  stages  of  development.  However,  by  and 
large,  these  models  have  not  been  as  widely  and  convincingly 
used  as  was  expected. 

In  this  paper  we  attempt  to  identify  the  causes  of  this  state 
of  affairs  and  suggest  some  remedial  actions.  For  exeimple,  we 
feel  that  very  often  the  models  are  used  without  a  clear  un¬ 
derstanding  of  their  underlying  assunptions  and  limitations. 
Also,  there  seems  to  be  some  misunderstanding  about  the  in¬ 
terpretations  of  model  inputs  and  outputs.  To  overcome  some 
of  these  difficulties,  we  provide  a  classification  of  the 
available  models  and  suggest  which  types  of  models  are  ap¬ 
plicable  in  a  given  phase  of  the  software  development  cycle. 

102)  TITLE:  GUIDE  FOR  MANAGING  NONDELIVERABLE  COMPUTER  RESOURCES 

AUTHOR:  MAJ.  GEORGE  W.  TREVER,  AFCMD/EPER,  KIRKLAND  AFB 

DCX:_DATE:  24  JAN  1984 

SOURCE:  RECEIVED  THRCXJGH  NS  LA 

ABSTRACT:  (OBJECTIVES)  This  guide  was  prepared  to  assist  managers 

structure  and  understand  policies  and  procedures  in  the  man¬ 
agement  of  nondeliverable  computer  resources  (NDCR) .  It  is 
the  intent  of  this  guide  to  capture  the  basic  management 
methods  and  lessons  learned  attributed  to  deliverable  embed¬ 
ded  conpater  systems  and  treunslate  them  into  nanagement  terms 
appropriate  for  NDCR. 

103)  TITLE:  METHODOLOGY  FOR  SOFTWARE  AND  SYSTEM  RELIABILITY  PREDICTION 

PHASE  II  INTERIM  REPORT 
ALTHOR:  J.  McCALL,  et  al. 

DOC_DATE:  MARCH  1985 

SOURCE:  RADC  F30602-83-C-0118 

ABSTRACT:  The  purpose  of  this  report  is  to  describe  the  interim  results  of 
a  research  emd  development  effort  to  develop  a  methodology  for 
predicting  emd  estimating  software  reliability.  This  report 
represents  interim  findings  during  Phase  II  of  the  project.  Thi 
effort  was  performed  under  Contract  Number  F30602-83-C-0118  for 
the  U.S.  Air  Force  Rome  Air  DevelofHi«nt  Center  (RADC). 
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The  ultimate  objective  of  this  effort  is  to  develop  characterization  tech¬ 
niques  for  predicting  TOTAL  SYSTEM  RELIABILITY  BASED  CW  MISSION  OR  OPERATING 
TIME.  The  techniques  must  include  the  combined  effects  of  BOTH  HARDWARE  AND 
SOFTVJARE  as  well  as  the  ENVIRONMEMT  in  which  the  system  operates.  The  purpose 
of  this  initial  survey  is  to  identify  those  factors  and  characteristics  which 
affect  the  SOFIVIARE  component  of  the  overall  system  reliability.  Specifically, 
the  survey  addresses: 

SYSTEM  SOFTWARE  RELIABILITY  -  The  probability  that  the  required 
software  will  perform  its  intended  functions  for  the  prescribed 
mission(s)  and  time  period(s)  in  the  specified  operating  env¬ 
ironment,  without  causing  system  outage  or  failure. 

It  should  be  noted  that  the  above  definition  does  not  specifically  address 
"qualities"  of  the  software  other  than  its  zdDility  to  perform  as  specified 
without  causing  overall  system  outage.  For  example,  it  is  possible  for  very 
inefficient  software  to  be  very  reliable.  The  intent  of  the  survey  is  to  cor¬ 
relate  qualitative  and  quantitative  factors.  THE  RATING  THAT  YOU  ARE  ASKED  TO 
PROVIDE  SHOULD  BE  BASED  CN  THE  FACTOR'S  RELATICWSHIP  TO  RELIABILITY,  NOT  ON 
ITS  IMPORTANCE  TO  NON-OPERATIONAL  QUALITIES. 

Horizontally,  the  survey  includes  an  overall  category  and  several  groupings  of 
categories  v^ich  are  typically  used  to  record  error  content  of  computer  pro¬ 
grams.  In  the  case  of  the  error  categories,  considerable  amount  of  QUANTI¬ 
TATIVE  data  is  available  from  past  projects.  Quantifying  the  overall  system 
reliability  is  the  goal  of  this  study. 

Vertically,  the  survey  includes  a  mixed  list  of  intrinsic  factors,  design  and 
development  methodologies,  techniques  and  operational  characteristics  which 
are  generally  well  covered  QUALITATIVELY  in  the  literature. 

You  are  asked  to  relate  the  rows  and  the  columns  by  marking  those  blocks  where 
you  feel  there  is  a  cause/effect  relationship  indicating  the  degree  of  cor¬ 
relation  by  using  the  codes  provided.  It  should  be  noted  that  some  of  the 
vertical  entries  such  as  "structured  design",  are  generally  assumed  to  have  a 
positive  effect  on  reliability  by  reducing  design  errors.  Others  such  as 
"predominately  real-time"  are  thought  to  decrease  reliability  by  increasing 
complexity.  In  this  survey,  you  are  not  asked  to  euialyze  the  type  of  cor¬ 
relation,  but  simply  indicate  the  degree  of  correlation. 

Although  this  survey  will  not  provide  quantitative  effects  of  the  qualitative 
factors,  it  is  expected  that  it  will  indicate  a  strong  relationship  between 
specific  factors  and  specific  error  types  which  in  turn,  can  be  related  to 
total  error  content  of  a  program.  Ttiis  total  error  content  can  then  be  com¬ 
bined  with  the  operational  "duty  cycling"  of  the  program  to  yield  a  quanti¬ 
tative  prediction  of  its  reliability. 

Definitions  which  are  unique  or  special  to  this  survey  are  presented  herein. 

A  general  glossary  of  software  terms  and  definitions  is  included  separately. 
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In  ocder  to  nvske  this  sutvey  ns  concise  ns  possible  and  to  insute  consistent 
interpretation  of  the  terminology  used,  the  following  definitions  are  pre¬ 
sented: 


SYSTEM  SOPIVIARE  RELIABILIl’Y  -  The  probability  that  the  reijiiiied  software  will 
perform  its  intended  functions  for  the  prescril^ed  mission(s)  and  time  ^ler- 
iod(s)  in  the  specified  operating  environment,  without  causing  system  outage 
or  failure. 

SW  SPECIFICATION  ERRORS  -  These  include  all  errors  resulting  from  missing, 
incorrect  or  misunderstood  requirements.  Include  in  this  category  any  problems 
associated  with  the  ccMnmvinication  of  requirements  between  the  contractor  and 
the  developer. 

SW  DESIGN  ERRORS  -  These  are  errors  which  occur  in  the  logical  implementation 
of  the  required  function.  Include  in  this  category  all  errors  where  a  required 
function  is  not  included,  not  invoked,  not  checked,  not  complete  or  does  not 
produce  correct  results  due  to  logical  construction. 

SW  CODING  ERRORS  -  These  are  computational  or  calculation  errors.  Included 
are:  errors  of  omission  such  as  unitialized  variables;  mathematical  errors 
such  as  incorrect  expressions,  conversion  and  truncation  errors;  and  progreun- 
ming  errors  such  as  improper  use  of  indices,  variables  and  overlays.  Since, 
this  survey  is  addressed  to  operational  reliability,  DO  NOT  INCLUDE  SYNTAX 
ERRORS  that  would  be  eliminated  prior  to  operation. 

SW  INTERFACE  PROBLEMS  -  This  category  addresses  all  errors  which  occur  between 
software  components  of  the  system  such  as  wtien  one  program  unit  fails  to  call, 
calls  in  the  wrong  sequence,  or  otherwise  improperly  calls  another  program 
unit.  It  also  includes  all  errors  resulting  from  the  improper  sharing  or 
passing  of  data  and/or  control  variables  between  program  units.  Examples 
include:  passing  wrong  arguments,  mismatched  scale  factors,  missing  arguments, 
etc. 

{{W  INT'ERFACE  PROBLEMS  -  This  category  includes  all  errors  which  result  in  loss 
of  data  or  untimely  exchange  of  data  between  system  hardware  and  embedded 
software.  Exeimples  include  situations  where  buffers  become  saturated  or  com¬ 
putation  cycles  exceed  their  timing  allocations.  Also  included  are  errors 
caused  by  improper  data  exchange  between  system  hardware  and  embedded  soft¬ 
ware.  Elxamples  include:  missing  data,  incorrect  data,  mismatched  scales,  etc. 

HUMAN  INTERFACE  PROBLEMS  -  This  category  includes  all  operator  errors  that  are 
not  corrected  or  compensated  for  by  the  computer  software.  Examples  include 
the  acceptance  of  improper  commands  and  data  and  the  rejection  of  proper  in¬ 
puts.  Also  included  are  output  errors  *vhich  result  in  human  interface  problems 
such  as  missing  output,  incorrect  output,  ambiguous  output,  etc. 
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CAPACITY  PROBLEMS  -  This  category  includes  all  errors  where  the  system  per¬ 
forms  all  of  its  operational  tasks  but  not  within  its  required  timing  con¬ 
straints  or  only  performs  them  on  a  subset  of  its  intended  input  domain,  i.e., 
there  are  situations  where  it  doesn't  vratk  due  to  the  quantity  or  content  of 
its  tasks.  For  example,  buffers  that  become  saturated  during  periods  of  high 
activity. 
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Thirty  surveys  were  distributed  and  23  were  returned.  Each  row  of  the  survey 
represented  a  software  factor  or  characteristic  which  affects  system  relia¬ 
bility.  The  first  column  of  the  survey  was  entitled  System  Reliability  Impact, 
cind  the  remaining  columns  represented  the  following  error  categories: 


A:  SW  Specification  Errors 
B:  SW  Design  Errors 
C:  SW  Coding  Errors 
D:  SW  Interface  Problems 
E:  HW  Interface  Problems 
F:  Human  Interface  Problems 
G:  Capacity  Problems 


The  participants  were  asked  to  assign  a  rating  to 
degree  of  correlation  between  the  factors  cind  the 
ticipant  had  no  opinion,  the  block  was  left  blank. 

each  block  indicating  the 
error  categories.  If  a  par- 

Each  of  the  four  possible  ratings  was  given  a  numerical  value  as  shown  below. 

Rating 

Numerical 

Value 

H 

(High  Correlation) 

9.0 

M 

(Medium  Correlation) 

5.0 

L 

(Low  Correlation) 

1.0 

0 

(No  Correlation) 

0.0 

A  blank  meant  that  the  respondee  had  no  opinion  on  that  particular  item  and 
so  blanks  were  ignored  in  the  analysis.  Using  the  numerical  values  for  the 
ratings,  overall  averages  were  calculated  for  each  block  of  the  survey;  i.e., 
for  each  row  and  column  combination.  Individual  averages  for  each  row  were 
computed  as  well  as  overall  row  averages.  Although  numerical  averages  were 
computed  for  the  qualitative  and  other  factors  listed  on  the  last  page  of  the 
survey,  they  were  not  included  in  the  overall  rankings. 

Table  D-1  shows  the  top  twenty  software  factors  ranked  in  decreasing  order  of 
overall  row  average.  It  is  obvious  that  specifications,  application  types  and 
design  methodologies  were  determined  to  be  the  leading  influences  on  system 
reliability.  Te±)le  D-2  gives  the  overall  average,  the  number  of  responses  and 
the  standard  deviation  for  each  block  of  the  survey. 


m 

TABLE  D-1 .  TWENTY 

HIGHEST  RANKED  SOFTWARE  ATTRIBUTES  /  FACTORS 

m 

Row  Average 

Attribute  /  Factor 

7.91 

Interface  Design  Spec 

m 

7.62 

Software  Requirements  Spec 

.•'‘Oh* 

7.41 

System  Requirements  Spec 

7.40 

Frequent  Walkthroughs 

7.14 

Software  Design  Spec 

c:^v 

6.83 

Definition/Enforcement  of  Stzuidards 

m 

6.46 

Predominantly  Real-Time 

rry- 

6.40 

Experience  Level  of  Team  Members 

1-;: 

«  •  . 

6.38 

Predominantly  Control 

Mi 

6.31 

Top-Down  &  Structured  Approaches 

6.24 

Test  Plans,  Procedures  &  Reports 

ifi 

5-:-' 

6.21 

Predominantly  Interactive 

9 

6.13 

Formal  Reviews  ( PDR , CDR , etc ) 

6.12 

Requirements  Traceability  Matrix 

•  ■  • 

h  V* 

6.10 

Modular  Construction 

[?i: 

6.05 

Performed  in  the  Field 

f-Cv 

5.98 

Structured  Design 

■t  - 

•  *  ■* 

5.91 

Constant  Mission  Usage 

« • 

5.87 

Number  of  Software  Interfaces 

S 

w.>- 

v>,> 

? 

5.83 

S/W  Development  Plan 

1 

• 
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TABLE  I>-2.  STATISTICAL  RESULTS  OF 

p  I  \n\ 

1'  SURVEY 

SW  FACTOR/CHARACTERISTIC 

Sys .  Re 1 
Impact 

A 

B 

C 

D 

E 

F 

G 

Predominantly  Control 

Average 

6.9 

7.7 

7.0 

5.0 

7.0 

5.9 

6.3 

4.7 

No.  Resp. 

19. 

18. 

18. 

L5. 

14. 

15. 

12. 

14. 

Std.  Dev. 

2.8 

2.4 

3.1 

3.7 

3.0 

3.9 

3.1 

3.9 

Predominantly  Computational 

Average 

4.1 

5.9 

6.9 

5.6 

4.6 

3.3 

1.7 

3.3 

No .  Resp . 

18. 

16. 

17. 

13. 

12. 

12. 

13. 

13. 

Std.  Dev. 

3.0 

3.5 

2.5 

3.6 

3.3 

3.8 

2.5 

3.6 

Predominantly  Input/Output 

Ave  rage 

5.2 

7.0 

6.3 

5.3 

6.5 

6.7 

3.6 

4.8 

No.  Resp. 

18. 

16. 

15. 

14. 

15. 

16. 

13. 

13. 

Std.  Dev. 

3.2 

3.3 

3.3 

3.3 

3.1 

2.7 

3.2 

3.1 

Predominantly  Real-Time 

Average 

7.0 

6.3 

6.9 

5.9 

6.7 

6.2 

4.9 

7.1 

No.  Resp. 

20. 

15. 

15. 

13. 

14. 

15. 

11. 

15. 

Std.  Dev. 

2.8 

3.4 

3.0 

3.7 

3.0 

3.5 

3.7 

2.6 

Predominantly  Interactive 

Ave  rage 

6.1 

7.5 

6.8 

5.3 

6.1 

5.1 

7.9 

4.2 

No.  Resp. 

18. 

13. 

13. 

13. 

15. 

14. 

18. 

12. 

Std.  Dev. 

3.0 

2.6 

3.1 

3.4 

3.2 

3.9 

2.7 

3.1 

Number  of  Hardware  Interfaces 

Average 

6.5 

4.9 

5.4 

3.2 

5 . 5 

8.6 

3.5 

4.2 

No.  Resp. 

21. 

14. 

16. 

14. 

13. 

19. 

15. 

14. 

Std.  Dev. 

3.0 

3.6 

3.0 

3.1 

3.7 

1.3 

3.1 

3.4 

Number  of  Software  Interfaces 

Average 

7.7 

7.4 

7.4 

5.0 

3.6 

3.0 

2.6 

3.3 

No.  Resp. 

21. 

15. 

15. 

14. 

19. 

14. 

14. 

14. 

Std.  Dev. 

2.3 

2.5 

2.5 

3.1 

1.3 

3.3 

2.2 

3.3 

Nufiiber  of  Human  Interfaces 

Average 

6.3 

5.5 

5.5 

3.2 

3.8 

3.2 

7.8 

2.9 

No.  Resp. 

19. 

15. 

15. 

14. 

14. 

13. 

17. 

13. 

Std.  Dev. 

3.0 

3.3 

3.3 

3.1 

3.0 

2.9 

2.7 

2.9 

Number  of  Functions  Performed 

Average 

6.9 

6.9 

6.7 

5.0 

7.0 

3.7 

3.4 

4.9 

No.  Resp. 

21. 

17. 

19. 

16. 

16. 

15. 

15. 

15. 

Std.  Dev. 

2.4 

2.9 

2.8 

3.3 

2.1 

3.5 

3.2 

2.9 

Overall  Program  Size 

Ave  rage 

5.9 

5.2 

6.1 

5.6 

5.5 

2.3 

2.0 

6.6 

No.  Resp. 

20. 

15. 

18. 

16. 

15. 

14. 

14. 

15. 

Std.  Dev. 

3.3 

3.6 

2.8 

2.9 

3.8 

3.3 

2.8 

2.0 

Number  of  Compilation  Units 

Average 

4.9 

4.5 

4.6 

4.7 

5.9 

1.6 

1.6 

2.9 

No.  Resp. 

18. 

15. 

18. 

16. 

16. 

14. 

14. 

14. 

Std.  Dev. 

3.2 

3.9 

3.1 

3.5 

3.3 

2.7 

2.7 
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SW  FACTOR/CHAFACTERISTIC 
Maximum  size  per  unit 

Number  of  Entries  and  Exits 

Number  of  Control  Vari2ibles 


Use  of  Single-Function 
Modules 


Number  of  Modules 


Maximum  Module  Size 


Hierarchical  Control  Between 
Modules 


Logical  Coupling  Between 
Modules 


Data  Coupling  between  Modules 


Separate  Design  and  Coding 


Independent  Test  Organization 


Independent  Quality  Assurance 


Sys.  Rel. 

Impact 

Average 

5.9 

No.  Resp. 

17. 

Std.  Dev. 

3.4 

Average 

7.7 

No.  Resp. 

18. 

Std.  Dev. 

2.4 

Average 

6.8 

No.  Resp. 

18. 

Std.  Dev. 

2.5 

Average 

6.8 

No.  Resp. 

18. 

Std.  Dev. 

3.1 

Average 

3.5 

No.  Resp. 

17. 

Std.  Dev. 

3.2 

Average 

5.7 

No.  Resp. 

18. 

Std.  Dev. 

2.8 

Average 

5.7 

No.  Resp. 

18. 

Std.  Dev. 

3.1 

Average 

6.5 

No.  Resp. 

16. 

Std.  Dev. 

3.2 

Average 

6.1 

No.  Resp, 

18. 

Std.  Dev. 

2.7 

Average 

3.8 

No.  Resp. 

17. 

Std.  Dev. 

3.5 

Average 

6.7 

No.  Resp. 

19. 

Std.  Dev. 

3.1 

Average 

7.0 

No.  Resp. 

18. 

Std.  Dev. 

2.8 

4.5  5.1 

14.  15. 
3.9  3.4 

5.0  7.3 

15.  19. 
4.2  2.6 

5.6  5.6 

14.  17. 
3.9  3.0 

5.1  7.1 

15.  17. 

3.7  3.0 

4.8  6.8 
13.  17. 

3.9  3.0 

4.9  6.5 
13.  18. 
4.6  2.9 

4.2  6.6 
13.  15. 

3.5  2.9 

4.4  7.1 

13.  15. 

4.2  3.0 

4.4  7.8 

14.  16. 

4.3  2.4 

3.6  6.3 
12.  18. 

3.2  3.1 

5.8  6.5 
13.  15. 

3.9  3.4 

6.2  5.7 
13.  16. 
3.6  3.4 


5.5  5.2 

15.  13. 

3.1  3.3 

6.7  8.4 
19.  19. 

2.8  1.5 

6.5  7.8 

16.  17. 

2.9  1.9 

5.6  6.6 

17.  15. 
3.0  2.9 

4.6  6.4 
14.  14. 

3.4  3.4 

6.2  4.6 
17.  14. 

3.1  3.5 

5.0  7.0 

13.  16. 

3.3  3.3 

5.9  7.4 

14.  17. 

3.2  2.5 

6.9  7.2 

15.  18. 
3.0  2.5 

5.8  5.3 

16.  14. 

3.3  2.9 

6.0  5.5 
19.  15. 
3.3  3.4 

5.9  4.9 
16.  14. 

3.5  3.6 


2.9  2.1  3.8 

14.  13.  13. 

3.4  2.9  3.6 

3.1  2.9  2.2 

15.  14.  13. 

3.6  3.7  3.3 

2.8  3.3  2.7 
14.  13.  14. 

3.2  3.2  2.9 

2.6  2.4  1.7 

13.  13.  13. 

2.8  3.4  3.3 

2.4  1.9  2.9 

14.  13.  14. 

3.3  2.7  3.0 

2.2  2.1  3.4 

14.  13.  14. 

3.4  3.4  3.2 

2.0  2.8  1.2 
13.  13.  13. 
3.0  3.4  2.4 

2.2  2.3  2.3 
13.  13.  13. 

2.9  2.8  3.5 

2.8  2.5  1.7 

15.  13.  13. 

3.3  3.0  2.8 

3.8  3.3  2.0 

12.  12.  10. 

3.6  2.8  2.1 

5.3  5.8  3.0 
12.  13.  11. 

3.3  3.5  2.9 

4.8  6.1  2.2 
12.  14.  11. 
3.6  3.4  2.9 
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Sys.  Rel. 

SW  FACTOR/CHARACTERISTIC 

Impact 

A 

B 

c 

D 

E 

F 

G 

Independent  Configuration 

Average 

5.7 

5.5 

5.5 

4.4 

4.6 

4.5 

4.5 

1.6 

Control 

No.  Resp. 

17. 

13. 

14. 

15. 

14. 

11. 

13. 

10. 

Std.  Dev. 

2.9 

3.8 

3.6 

3.7 

3.5 

3.4 

4.0 

1.8 

Independent  Verification 

Average 

6.1 

6.3 

6.6 

6.3 

5.7 

5.2 

5.8 

2.3 

and  Validation 

No.  Resp. 

18. 

12. 

15. 

15. 

15. 

12. 

13. 

10. 

Std.  Dev. 

3.3 

3.3 

3.3 

3.6 

3.6 

3.4 

3.5 

3.0 

Progranoning  Team  Structure 

Average 

5.4 

3.6 

5.2 

5.7 

4.1 

2.2 

3.3 

2.0 

No.  Resp. 

18. 

14. 

17. 

17. 

15. 

11. 

12. 

10. 

Std.  Dev. 

3.0 

3.9 

3.0 

2.9 

2.5 

2.3 

2.8 

2.1 

Educational  Level  of  Team 

Average 

4.5 

4.3 

5.2 

3.8 

4.9 

3.8 

4.5 

2.4 

Members 

No.  Resp. 

17. 

14. 

17. 

16. 

14. 

12. 

13. 

10. 

Std.  Dev. 

2.5 

3.3 

3.3 

3.2 

2,8 

3.2 

3.3 

2.3 

Experience  Level  of  Team 

Average 

7.9 

6.1 

7.4 

6.0 

6,6 

5.3 

5.8 

4.9 

Members 

No.  Resp. 

18. 

16. 

17. 

16. 

14. 

12. 

13. 

10. 

Std.  Dev. 

1.8 

3.7 

2.8 

3.4 

2.8 

2.8 

3.1 

3.9 

Definition/Enforcement  of 

Average 

7.3 

6.8 

7.1 

7.4 

7.8 

7.2 

6.5 

3.1 

Standards 

No.  Resp. 

19. 

13. 

17. 

17. 

14. 

13. 

13. 

10. 

Std.  Dev. 

2.4 

3.5 

2.5 

2.5 

2.7 

2.6 

3.5 

3.1 

Use  of  High  Order  Language 

Average 

6.9 

3.1 

3.5 

8.4 

5.8 

4.0 

2.2 

4.4 

(HOD 

No.  Resp. 

19. 

14. 

15. 

19. 

15. 

13. 

13. 

12. 

Std.  Dev. 

2.8 

4.1 

3.7 

2.0 

3.8 

3.8 

2.8 

3.5 

Formal  Reviews  (PDR,CDR,etc) 

Average 

5,4 

7.8 

6.9 

5.1 

6.4 

7.1 

6.6 

3.4 

No.  Resp. 

21. 

17. 

19. 

16. 

16. 

13. 

14. 

12. 

Std.  Dev. 

3.1 

2.4 

2.8 

4.0 

3.5 

3.3 

3.7 

3.4 

Frequent  Walkthroughs 

Average 

7,5 

7.7 

8.2 

7.6 

8.0 

7.5 

7.5 

4.3 

No.  Resp. 

19. 

15. 

19. 

18. 

17. 

13. 

14. 

12. 

Std.  Dev. 

2.4 

2.5 

2.1 

3.2 

2.4 

2.0 

2.7 

4.0 

Top-Down  &  Structured 

Average 

6.8 

6.8 

7.4 

7.1 

7.9 

5.2 

5.2 

2.6 

;^proaches 

No.  Resp. 

20. 

16. 

18. 

17. 

15. 

13. 

13. 

12. 

Std.  Dev. 

2.7 

2.9 

2.8 

2.9 

2.4 

3.3 

3.7 

3.5 

Unit  Development  Folders 

Average 

6.2 

4.2 

6.9 

7.0 

6.3 

4.9 

5.0 

2.8 

No.  Resp. 

17. 

15. 

16. 

16. 

15. 

13. 

13. 

13. 

Std.  Dev. 

2.4 

3.6 

2.7 

2.5 

2.5 

3.4 

3.3 

3.0 

Software  Development  Library 

Average 

6.5 

5.2 

6.1 

7.2 

6.2 

3.9 

3.6 

2.3 

No.  Resp. 

16. 

12. 

14. 

16. 

13. 

12. 

11. 

12. 

Std.  Dev. 

3.2 

4.2 

3.0 

3.4 

3.6 

3.6 

3.9 

2.5 
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SW  FACTOR/CHARACTEKISTIC 

Sys.  Rel. 
Impact 

A 

B 

c 

D 

E 

F 

G 

Formal  Change  &  Error 

Average 

6.5 

5.5 

5.7 

5.9 

5.5 

4.6 

4.7 

2.1 

Reporting 

No.  Resp. 

16. 

13. 

15. 

16. 

13. 

12. 

13. 

13. 

Std.  Dev. 

2.0 

3.8 

3.2 

3.5 

2.9 

2.8 

3.0 

2.9 

Progress  &  Status  Reporting 

Average 

4.7 

3.9 

5.0 

4.7 

4.6 

3.9 

4.6 

1.5 

No.  Resp. 

15. 

13. 

16. 

16. 

11, 

11, 

11. 

12. 

Std.  Dev. 

3.6 

3.9 

3.9 

3.8 

3.3 

3.1 

3.3 

2.2 

Modular  Construction 

Average 

8.4 

4.2 

8.1 

7.4 

8.2 

4.3 

4,1 

1.8 

No.  Resp. 

19. 

14. 

17. 

15. 

15. 

12. 

13, 

12. 

Std.  Dev. 

2.0 

4.0 

2.7 

2.9 

2.2 

3.0 

3.6 

2.0 

Structured  Design 

Average 

8.0 

3.9 

7.9 

7.1 

7,3 

4.9 

3.6 

2.3 

NO.  Resp. 

20. 

15. 

19. 

15. 

14. 

11. 

11. 

12. 

Std.  Dev. 

2.2 

4.1 

2.6 

2.6 

2.6 

3.2 

3.4 

2.9 

Structured  Code 

Average 

7.7 

3.2 

4.8 

8.1 

7.3 

4.5 

3.4 

2.2 

No.  Resp. 

19. 

14. 

15. 

18. 

14. 

12. 

12. 

13. 

Std.  Dev. 

2.3 

4.0 

3.9 

2.2 

3.0 

3.8 

3.4 

2.8 

Flow  Charts 

Average 

4.6 

4.1 

5,9 

6.1 

6.8 

4.6 

3.2 

2.4 

No.  Resp. 

18. 

14. 

13. 

15. 

13. 

11. 

12. 

11. 

Std.  Dev. 

3.3 

4.1 

2.4 

2.8 

2.6 

2.8 

2.9 

3.1 

Structure  Charts 

Average 

6.0 

5.6 

7.0 

5.7 

6.7 

3.8 

3.5 

2.0 

No.  Resp. 

16. 

11. 

14. 

12. 

12. 

10. 

11. 

11. 

Std.  Dev. 

3.1 

3.6 

2.1 

3.3 

2.7 

3.3 

3.8 

3.0 

Decision  Tables 

Average 

5.6 

5.7 

6.7 

5.7 

6.2 

3.5 

2.9 

2.2 

No.  Resp. 

13. 

10. 

12. 

12. 

10. 

11. 

10. 

10. 

Std.  Dev. 

2.2 

3.3 

2.7 

3.3 

3.3 

2.8 

2.9 

3.1 

HIPO  Charts 

Average 

5.5 

6.0 

7.3 

5.7 

6.1 

4.2 

3.1 

2.4 

No.  Resp. 

16. 

11. 

14. 

12. 

11. 

10. 

11. 

11. 

Std.  Dev. 

2.5 

2.8 

2.1 

3.3 

3.1 

3.2 

3.4 

3.6 

System  Requirement  Spec 

Average 

7,7 

8.8 

7.9 

5.6 

7.7 

8.0 

7.4 

5.5 

No.  Resp. 

19. 

18. 

15. 

13. 

12. 

12. 

13. 

13. 

Std.  Dev. 

2.3 

0.9 

2.4 

3.6 

2.6 

2.5 

2.8 

3.8 

Software  Requirement  Spec 

Average 

7.8 

9.0 

8.5 

6.9 

8.7 

5.9 

7.4 

5.6 

No.  Resp. 

17. 

17. 

16. 

15. 

14. 

12. 

13. 

11. 

Std.  Dev. 

2.4 

0.0 

1.4 

3.3 

1.1 

3.6 

2.8 

3.6 

Interface  Design  Spec 

Average 

8.3 

7.7 

8.8 

7.0 

8.7 

9.0 

8.0 

4.8 

No.  Resp. 

18. 

15. 

17. 

14. 

15. 

13. 

13. 

11. 

Std.  Dev. 

1.5 

2.5 

1.0 

3.4 

1.0 

0.0 

2.6 

3.8 

D  -  7 
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SW  FACTOR/CHARACTERISTIC 

Sys.  Rel 
Impact 

A 

B 

C 

D 

E 

F 

G 

Software  Design  Spec 

Average 

7.4 

7.5 

8.3 

7.1 

7.6 

6.3 

6.4 

5.8 

No.  Resp. 

18. 

15. 

18. 

15. 

15. 

12. 

13. 

12. 

Std.  Dev. 

2.9 

3.2 

2.3 

3.1 

2.7 

2.8 

3.4 

3.7 

Source  Listings 

Average 

4.2 

3.8 

5.8 

7.4 

5.5 

3.0 

3.8 

2.2 

No.  Resp. 

17. 

13. 

13. 

17. 

13. 

11. 

12. 

11. 

Std.  Dev. 

3.3 

4.0 

3.9 

2.8 

3.3 

2.9 

4.1 

2.9 

Test  Plans,  Procedures  & 

Average 

7.3 

6.4 

6.7 

6.3 

6.4 

5.6 

5.4 

5.2 

Reports 

No.  Resp. 

19. 

14. 

14. 

18. 

14. 

13. 

14. 

13. 

Std.  Dev. 

2.0 

3.5 

3.0 

3.1 

2.5 

3.2 

3.3 

3.2 

S/W  Development  Plan 

Average 

6.5 

6.3 

6.8 

6.8 

6.7 

4.3 

5.2 

2.7 

No.  Resp. 

21. 

15. 

16. 

18. 

14. 

12. 

13. 

12. 

Std.  Dev. 

3.0 

3.3 

2.5 

2.5 

2.6 

3.4 

3.7 

3.4 

S/W  Quality  Assurance  Plan 

Average 

5.4 

5.4 

6.3 

5.8 

5.5 

3.9 

5.1 

2.5 

No.  Resp. 

18. 

15. 

15. 

16. 

15. 

13. 

15, 

13. 

Std.  Dev. 

3.1 

3.9 

3.3 

3.3 

3.3 

3.1 

3.7 

3.8 

S/W  Configuration  Management 

Average 

5.8 

4.6 

5.2 

5.2 

4.9 

3.0 

3.8 

1.9 

Plan 

No.  Resp. 

16. 

14. 

13. 

15. 

13. 

11. 

12. 

12. 

Std,  Dev. 

2.6 

3.5 

3.6 

3.6 

3.4 

2.9 

3.2 

2.9 

Requirements  Traceability 

Average 

7.2 

8.3 

7.5 

4.9 

6.3 

4.6 

5.5 

3.0 

Matrix 

No.  Resp. 

18. 

17. 

16. 

13. 

12. 

12. 

12. 

13. 

Std.  Dev. 

2.5 

1.6 

2.0 

3.4 

2.0 

3.3 

3.6 

3.9 

Version  Description  Document 

Average 

3.6 

4.8 

4.3 

4.8 

3.8 

2.8 

3.5 

1.6 

No.  Resp. 

15. 

13. 

12. 

13. 

13. 

12. 

12. 

12. 

Std,  Dev. 

2.6 

3.8 

3.4 

3.8 

3.0 

2.9 

3.7 

2.7 

Software  Discrepeuicy  Reports 

Average 

6.3 

5.2 

6.3 

7.1 

5.6 

4.7 

4.9 

3.5 

No.  Resp. 

18. 

13. 

14. 

15. 

13. 

13. 

13. 

13. 

Std.  Dev, 

3.1 

4.0 

3.3 

2.6 

3.6 

3.4 

3.8 

4.0 

Constant  Mission  Usage 

Average 

6.7 

6.3 

6.3 

5.6 

6.1 

7.3 

5.3 

2.8 

No.  Resp, 

18. 

12. 

12. 

12. 

11. 

12. 

12. 

10. 

Std.  Dev. 

3.3 

3.7 

3.1 

3.5 

3.6 

2.7 

3.3 

3.6 

Periodic  Mission  Usage 

Average 

4.9 

4.8 

4.5 

3.5 

4.6 

5.3 

3.7 

1.9 

No.  Resp. 

16. 

11. 

11. 

11. 

10. 

12. 

11. 

10. 

Std.  Dev. 

2.6 

3.4 

3.4 

2.2 

2.3 

2.7 

2.8 

2.9 

Infrequent  Mission  Usage 

Average 

4.7 

3.8 

4.3 

3.2 

3.4 

4.7 

3.5 

1.5 

No.  Resp. 

18. 

12. 

12. 

12. 

10. 

12. 

12. 

10. 

Std.  Dev. 

D  -  8 

3.9 

4.0 

3.8 

3.3 

3.4 

3.6 

3.7 

2.7 
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SW  FACTOIV'CHARACTERISTIC 

Sys.  Rel 
Impact 

A 

B 

C 

D 

S 

F 

G 

Variability  of  Hardware 

Average 

5.9 

4.8 

4.4 

2.4 

2.8 

8.0 

4.6 

1 

.4 

No.  Resp. 

17. 

9.0 

10. 

10. 

9.0 

12. 

10. 

7 

.0 

Std.  Dev. 

2.7 

4.3 

3.7 

3,0 

2.1 

1.8 

3.0 

1 

.6 

Training  Level  of  Operators 

Average 

5.4 

2.5 

3.4 

2.3 

1.6 

2.9 

6.3 

1 

.1 

No.  Resp. 

18. 

11. 

10. 

11. 

10. 

9.0 

12. 

8 

.0 

Std.  Dev. 

2.7 

3.8 

3.2 

3.2 

2.4 

3.2 

3.3 

1 

.6 

Varie±>ility  of  Input  Data 

Average 

6.3 

4.1 

5.6 

4.5 

6.0 

3.8 

5.7 

3 

.1 

No.  Resp. 

18. 

12. 

11. 

12. 

12. 

11. 

11. 

9 

.0 

Std.  Dev. 

2.7 

3.6 

3.6 

3.8 

2.5 

3.7 

3.0 

3 

.6 

Variedoility  of  Outputs 

Average 

4.1 

3.8 

4.9 

4.5 

4.9 

3.8 

4.6 

0 

.9 

No.  Resp. 

16. 

12. 

10. 

12. 

11. 

10. 

11. 

8 

.0 

Std.  Dev. 

3.5 

3.7 

3.9 

3.8 

3.7 

3.3 

2.8 

0 

.4 

Degree  of  Hunan  Interaction 

Average 

7.0 

3.6 

4.0 

2.8 

3.3 

4,6 

7.8 

0 

.6 

No.  Resp. 

18. 

11. 

10. 

12. 

10. 

9.0 

16. 

8 

.0 

Std.  Dev. 

2.5 

3.4 

3.4 

2.9 

2.2 

2.4 

1.9 

0 

.5 

Training  Exercises 

Average 

3.8 

2.1 

2.2 

2.5 

2.1 

2.4 

3.8 

0 

.3 

No.  Resp. 

17. 

11. 

10. 

11. 

10. 

11. 

12. 

9 

.0 

Std.  Dev. 

2.4 

3.4 

3.1 

3.0 

3.1 

3.6 

3.6 

0 

.5 

Periodic  Self  Test 

Average 

5.9 

4.3 

4.9 

3.3 

3.6 

3.4 

2.1 

0 

.6 

No.  Resp. 

16. 

10. 

9.0 

10. 

9.0 

11. 

9.0 

9 

.0 

Std.  Dev. 

2.9 

3.9 

3.6 

3.5 

3.6 

3.4 

2.2 

0 

.5 

Built-in  Diagnostics 

Average 

7.3 

4.0 

6.0 

5.3 

5.3 

4.8 

4.4 

1 

.4 

No.  Resp. 

19. 

11. 

11. 

12. 

10. 

12. 

10. 

9 

.0 

Std.  Dev. 

2.6 

3.3 

3.3 

3.7 

4.1 

3.6 

4.2 

2 

.9 

Performed  in  the  Field 

Average 

6.4 

6.4 

6.5 

5.8 

6.3 

6.7 

6.2 

3 

.3 

No.  Resp. 

16. 

8.0 

8.0 

10. 

9.0 

9.0 

9.0 

7 

.0 

Std.  Dev. 

3.7 

3.9 

3.0 

3.2 

2.8 

3.2 

3.7 

3 

.1 

Performed  at  Depot 

Average 

6.1 

4.9 

4.4 

5.0 

4.9 

5.4 

4.9 

1 

.7 

No.  Resp. 

14. 

8.0 

7.0 

8.0 

8.0 

8.0 

8.0 

6 

.0 

Std.  Dev. 

3.0 

3.9 

3.6 

3.0 

3.9 

4.1 

3.9 

1 

.6 

Performed  at  Factory 

Average 

5.5 

5.4 

4.0 

4.6 

4.0 

4.4 

4.9 

2 

.1 

No.  Resp. 

15. 

8.0 

8.0 

10. 

9.0 

9.0 

9.0 

7 

.0 

Std.  Dev. 

4.1 

4.1 

3.5 

3.5 

4.0 

3.8 

4.1 

2 

.0 

Correctness 

Average 

8.5 

9.0 

8.7 

8.3 

8.6 

7.8 

6.5 

2 

.5 

No.  Resp. 

17. 

13. 

12. 

11. 

10. 

10. 

10. 

8 

.0 

Std.  Dev. 

1.3 

0.0 

1.2 

1.6 

1.3 

2.7 

3.6 

4 

.0 

SW  FACTOR/CHARACTERISTIC 

Sys.  Rel. 
Impact 

A 

B 

c 

D 

E 

F 

G 

Validity 

Average 

8.4 

7.8 

8.6 

8.6 

9.0 

8.0 

6.4 

2.1 

No.  Resp. 

14. 

11. 

11. 

9.0 

8.0 

8.0 

8.0 

7.0 

Std.  Dev. 

1.5 

2.9 

1.2 

1.3 

0.0 

2.8 

3.9 

3.5 

Generality 

Average 

3.7 

6.2 

7,0 

7.0 

5,6 

6.7 

4.4 

1.6 

No.  Resp. 

14. 

10. 

8.0 

8.0 

7.0 

7.0 

8.0 

8.0 

Std.  Dev. 

3.5 

3.3 

3.0 

2.1 

3.6 

3.1 

3.5 

2.1 

Testability 

Average 

7.7 

6.0 

6.1 

6.7 

7.2 

5.8 

5.9 

1.8 

No.  Resp. 

19. 

11. 

11. 

12. 

9.0 

10. 

8.0 

8.0 

Std.  Dev. 

2.3 

3.8 

3.1 

3.2 

2.9 

3.2 

3.8 

2.1 

Efficiency  /  Economy 

Average 

3.3 

4.8 

5.3 

6.3 

5.8 

4.9 

3.4 

4.9 

No.  Resp. 

14. 

11. 

10. 

12. 

9.0 

9.0 

9.0 

8.0 

Std.  Dev. 

3.6 

4.2 

4.1 

3.7 

3.5 

3.6 

3.1 

3.9 

Resilience  (Robustness) 

Average 

7.1 

8.0 

8.1 

7.0 

7.3 

5.6 

5.4 

1.1 

No.  Resp. 

13. 

8.0 

9.0 

8.0 

7.0 

7.0 

7.0 

7.0 

Std.  Dev. 

2.8 

2.8 

2.7 

3.0 

3.1 

3.6 

3.8 

1.8 

Useaibility 

Average 

5.7 

5.7 

7.0 

6.3 

6.9 

5.6 

6.4 

2.4 

No.  Resp. 

14. 

9.0 

10. 

9.0 

8.0 

7.0 

8.0 

7.0 

Std.  Dev. 

3.8 

4.2 

3.4 

2.8 

3.3 

2.8 

3.2 

2.4 

Fault  Tolerance 

Average 

8.3 

8.2 

7.9 

6.6 

7.5 

5.4 

6.2 

2.3 

No.  Resp. 

17. 

10. 

11. 

10. 

8.0 

9.0 

9.0 

8.0 

Std.  Dev. 

1.6 

2.5 

1.9 

2.8 

3.0 

3.1 

3.7 

2.3 

Clarity 

Average 

7.1 

6.9 

7.3 

7.5 

7.7 

5.9 

7.3 

2.1 

No.  Resp. 

15. 

12. 

12. 

11. 

9.0 

8.0 

10. 

8.0 

Std.  Dev. 

2.1 

2.9 

2.7 

2.7 

2.8 

3.8 

3.1 

2.4 

Readability 

Average 

5.5 

5.6 

6.8 

7.3 

7.2 

6.0 

6.3 

1.6 

No.  Resp. 

15. 

11. 

11. 

12. 

9.0 

8.0 

9.0 

8.0 

Std.  Dev. 

2.6 

4.1 

3.7 

3.2 

3.5 

3.5 

3.5 

2.1 

Maintainability 

Average 

8.1 

5.6 

8.0 

8.0 

7.7 

5.4 

7.1 

3.3 

No.  Resp. 

17. 

11. 

12. 

12. 

9.0 

9.0 

9.0 

8.0 

Std.  Dev. 

2.2 

3.6 

1.8 

1.8 

2.8 

3.1 

3.2 

3.2 

ModifiaQaility 

Average 

7.1 

6.0 

7.2 

6.8 

7.0 

5.0 

6.9 

3.0 

No.  Resp. 

15. 

10. 

11. 

11. 

8.0 

7.0 

8.0 

7.0 

Std.  Dev. 

3.3 

3.7 

2.8 

2.8 

3.0 

2.3 

3.3 

3.4 

Flexibility 

Ave  rage 

6.4 

5.2 

7.8 

7.0 

7.5 

5.0 

5.9 

3.6 

No.  Resp. 

14. 

10. 

10. 

10. 

8.0 

7.0 

8.0 

7.0 

Std.  Dev. 

3.4 

4.2 

2.7 

2.8 

3.0 

3.3 

3.8 

4.1 

D  -  10 
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SW  FACTOR/CHABACTERISTIC 

Sys.  Rel. 
Impact 

A 

B 

Portability 

Average 

3.4 

5.3 

4.9 

No.  Resp. 

13. 

10. 

10. 

Std.  Dev. 

3.2 

3.7 

3.4 

Reusability 

Average 

3.6 

5.8 

5.3 

No.  Resp. 

12. 

9.0 

9.0 

Std.  Dev. 

3.2 

3.5 

3.3 

Interoperability 

Average 

4.6 

6.5 

5.9 

No.  Resp. 

14. 

10. 

8.0 

Std.  Dev. 

3.4 

3.6 

3.8 

D  -  11 


C  D  E  F  G 


6.6  5.5  4.9  3.4  3.9 
10.  8.0  9.0  8.0  8.0 

3.4  4.0  3.6  3.1  3.7 

5.9  6.1  4.5  4.3  3.7 
9.0  7.0  8.0  7.0  7.0 

2.7  3.0  2.6  3.0  3.2 

5.4  6.7  5.5  4.9  3.6 
8.0  7.0  8.0  8.0  7.0 

3.5  3.1  3.3  3.2  4.1 


SYSTEM 


RELIABILITY 


SURVEY 


Ccjnducted  By: 

Ed  Soistman  (MP-306) 
Martin  Marietta  Aerospace 
Orlando  Division 
P.O.  Box  5837 
Orlando,  FL  32855 

(305)  356-7062 


Conducted  For: 

Rome  Air  Development  Center 
Mr.  Gene  Fiorentino  (RADC/RBET) 
Griffiss  AFB,  New  York  13441 


Please  check  the  block  vAiich  MOST  closely  identifies  your  primary  involvement 
with  systems  involving  software: 

_  Systems  User 

_  Systems  Procurement 

_  Systems  Validation 

_  Systems  Operations 

_  Systems  Research 

OlHER: 


How  many  years  of  experience  do  you  have  in  systems  or  software  actWities? 

V«lhat  is  your  highest  level  educational  degree? _ 

What  single  discipline  best  describes  your  educatiorVexpertise:  _ 

Comnents  concerning  thus  survey:  _ 


Systems  Definition 
Software  Design 
Systems  Development 
Interns  Management 
Training  /  Education 


Address: 


Ccnpleted  By: 
Name; 


SHEET  1 


Please  rank  the  irajor  categories  in  order  of  inportance  to  system  reliability  using  the  values  1 
thru  5  with  "1"  being  the  assigned  to  the  most  inportant  category.  Use  a  similiar  ranking  within 
e?ich  of  the  categories  to  indicate  each  requirement's  likelihood  to  introduce  errors.  As  before, 
lise  a  value  of  "1"  to  indicate  the  most  likely. 


*  RANK  * 


*  SUB-  * 

*  RANK  * 


OPERATICNAL  APPLICATICN  TYPE 

Predominantly  Control 
Predominantly  Real  Time 
Predominantly  Irput/Output 
Predominantly  Interactive 
Predominantly  Corputatiaial 

mSSICN  VARIABILITy 

Many  Distinct  Operational  Missions 
Several  Variations  of  Operational  Missions 
Single  Operationad  Mission 


FUNCnCNAL  CCMPLEXITy 

Many  Operations  Required 
Many  Operations  Required 
Few  Operations  Required 
Few  (^rations  Required 

SYSTEM  INTCBACTTON 


Highly  Corplex 
Relatively  Simple 
Hiig^dy  Corplex 
Relatively  Sinple 


Extensive  Hardware  Interface  Requirements 
MinLial  Hardware  Interface  Requirements 
Extensive  Software  Interface  Requirements 
Minimal  Software  Interface  Requirements 
Extensive  Hunan  Interface  Requirements 
Minimal  Hunan  Interface  Requirements 

INPUT  DOMAIN  VARIABILITY 

Wide  Range  of  Error-Prone  Inputs 
wide  Range  of  Error-Free  Inputs 
Narrcv  Range  of  Error-Prone  Inputs 
Narrow  Range  of  Error-Free  Inputs 


SHEET  2 


Use  the  following  codes  to  indicate  the  relative  quantity  of  errors  introduced  during 


each  of  the  phases  shewn; 


(blank)  for  NO  OPINICW 

L  for  a  LOW  level  of  errors  introduced 
M  for  a  MODERATE  level  of  errors  introduced 
H  for  a  HIGH  level  of  errors  introduced 


*****  Phase  When  Introduced  ***** 


OPERATTCNAL  APPLICATTCN  TYPE 


Prelim.  Detailed 
Design  Design 


Predcminantly  Control 
Predominantly  Real  Time 
Predcminantly  Ir^t/Output 
Predominantly  Interactive 
Predominantly  Corputational 


MISSION  VARIABILITY 


Many  Distinct  Operational  Missions 
Several  Variations  of  Operational  Missions 
Single  Operational  Mission 


FLNCTTGNAL  COMPLEXITY 


Many  Operations  Required 
Many  Operations  Required 
Few  Operations  Required 
Few  Operations  Required 


Highly  Conplex 
Relatively  Sinple 
Hic^ily  Complex 
Relatively  Sinple 


SYSTEM  INIERACnCN 


Extensive  Hardware  Interface  Requirements 
Minimal  Hardware  Interface  Requirements 
Extensive  Softwire  Interface  Requirements 
Minimal  Software  Interface  Requirements 
Extensive  Hunan  Interface  Requirements 
Mininal  Human  Interface  Requirements 


INPUT  DOMAIN  VARIABILITY 


Wide  Range  of  Error-Prone  Irputs 
Wide  Range  of  Error-Free  Inputs 
Narrow  Range  of  Error-Prone  Inputs 
Narrow  Range  of  Error-Free  Inputs 


E  -  4 


SHEET  3 


Use  the  following  numeric  codes  to  indicate  the  percentage  of  inherent  errors  v^ich  fall  into 
each  category.  TOie  sum  of  each  rcw  should  equal  10  which  represents  100%  of  the  errors  induced. 


0%  of  Errors  Present 
10%  of  Errors  Present 


9  90%  of  Errors  Present 

10  100%  of  Errors  Present 


*****  General  Error  Category  ***** 
Logic  Inter-  I/D  Ccnp. 


OPERATICNAL  APPLICATICN  TYPE 


Predominantly  Control 
Predominantly  Real  Time 
Predominantly  Ir^t/tXitput 
Predominantly  Interactive 
Preciominantly  Coqputational 


raSSICN  VAlRIABILITY 


Many  Distinct  Operational  Missions 
Several  Variations  of  Operational  Missions 
Single  Operational  Mission 


FUNCnCNAL  RBaUIREMENTS 

Many  Functions  Required 
Many  Functions  Required  ■ 
Few  Functions  Required  - 
Few  Functions  Required  - 


-  Highly  Interrelated 

-  Relatively  Independent 
Highly  Interrelated 
Relatively  Independent 


SYSTEM  INIEPACnCN 


Extensive  Hardware  Interfacre  Requirements 
Minimal  Hardware  Interface  Requirements 
Extensive  Softvrare  Interface  Requirements 
Minimal  Software  Inter fac:e  Requirements 
Extensive  Human  Interfacre  Requirements 
Minimal  Hunan  Interface  Requirements 


INPOT  DOMAIN  VARIABILITY 


Wide  Range  of  Error-Prone  Irputs 
Wide  Range  of  Error-Free  Inputs 
Narrow  Range  of  Error-Prone  Inputs 
Narrow  Range  of  Error-Free  Inputs 


E  -  5 


■  V  .V 


■V  .  .. 


SHEET!  4 


Use  the  following  numeric  codes  to  indicate  the  relative  percentage  of  errors  that  mi^t  be 
avoided  by  use  of  the  listed  mechanism:  NOTE:  THE  SUM  DOES  NOT  HAVE  TO  BE  100%. 

(blank)  NO  OPINION 
0  0%  Error  Avoidance 

1  10%  Error  Avoidance 

9  90%  Error  Avoidance 

10  100%  Error  Avoidance 


OBGANIZATICNAL  OCNSIDESATICNS 

Independent  Quality  Assurance  Organization 

Independent  Test  Organization 

Independent  Verification  and  Validation  (IV&V) 

Use  of  a  Software  Si^jport  Library 

Use  of  a  Software  Configuration  Control  Board 

DOCUMENTATTON 

Thorough  and  Enforced  Software  Development  Plan 
Rigidly  Controlled  System  Requirements  Spec 
Rigidly  Controlled  Interface  Design  Spec 
Rigidly  Controlled  Software  Requirements  Spec 
Rigidly  Controlled  Software  Functional  Design  Spec 
Rigidly  Controlled  Software  Detailed  Design  Spec 

METHODS  EMPDOYED 

Requirements  Traceability  Matrix 
Structured  Analysis  Tools 
Program  Specification  Language  (PSL) 

Program  Design  Language  (PDL) 

High  Order  Language  (HOL) 

Hierarchical,  Top-Dcwn  Design 
Structured  Design 
Single  Function  Modularization 
Structured  Code 

Use  of  Automatic  Measurement  Tools 
Use  of  Automatic  Test  Tools 


*****  General  Error  Category  ***** 
Logic  Inter-  l/tO  Conp. 
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SHEETT  5 


Use  the  following  numeric  codes  to  indicate  the  relative  percentage  of  errors  that  might  be 
detected  by  use  of  the  listed  mechanism:  NOTC:  TOE  SUM  DOES  NOT  HAVE  TO  BE  100%. 

(blank)  NO  OPINICN 
0  0%  Error  Detection 

1  10%  Error  Detection 

9  90%  Error  Detection 

10  100%  Error  Detection 


INTERNAL  REVIEWS 

Frequent  Peer  Walkthroughs 
Infrequent  Peer  Walkthroughs 
Frequent  Progress  Reviews 
Infrequent  Progress  Reviews 
Frequent  Quality  Audits 
Infrequent  Quality  Audits 


*****  General  Error  Category  ***** 
Logi '  Inter-  I/O  Ccrp. 


ERROR  HANDLING 

Use  of  Software  Problem  Reports  Prior  to  PQT 
Use  of  Software  Problem  Reports  Subsequent  to  P(?r 
Use  of  Software  Problem  Reports  Subsequent  to  PQI 
Use  of  Specification  Change  Notices  (SCN's) 

Use  of  Engineering  Change  Notices  (EOJ's) 


FORMAL  REVIEWS 

Software  Requirements  Review  (SRR) 
Preliminary  Design  Review  (PDR) 
Critical  Design  Review  (CDR) 

Test  Readiness  Review  (TOR) 

Functional  Configuration  Audit  (FCA) 
Physical  Configuration  Audit  (PCA) 

TESTS  AND  DEJOCTRATICNS 

Informed  Uhit-Level  Testing 
Preliminary  Quedification  Testing  (PQT) 
Formal  Qualification  Testing  (FQT) 
Software  Integration  Testing 
System  Integration  Testing 
Operational  Field  Testing 
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INSTRUCTIONS 


C,  L  0  S  S  A  R  Y 


FOR 

SYSTEM  RELIABILITY  SURVEY 


This  instruction  packet  is  provided  to  explain  and  clarify  the  enclosed  survey 
form.  It  should  be  used  as  a  reference  whenever  the  terms  or  instructions 
used  in  the  survey  need  further  clarification. 

There  is  a  separate  section  for  each  of  the  five  sheets  of  the  survey  for  ease 
of  reference.  Each  section  contains  an  expanded  set  of  instructions,  defin¬ 
itions  of  each  of  the  terms  used  as  column  headings  and  definitions  for  each 
of  the  row  entries. 

Although  most  definitions  apply  to  more  than  one  survey  sheet,  the  definitions 
have  been  repeated  for  ease  of  use. 


*  * 

*  SPECIAL  NarE  FOR  SHEETS  1,  2  AND  3  * 

*  * 
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In  any  large  software  product,  it  is  extremely  difficult  to  precisely 
identify  inherent  characteristics  of  the  overall  product  due  to  the 
vast  number  and  diversity  of  functional  requirements  that  must  be  sat¬ 
isfied. 

Your  responses  to  the  survey  questions  should  be  based  on  the  assump¬ 
tion  that  any  software  system  can  be  decomposed  into  functionally  dis¬ 
crete  sets  of  requirements  which,  in  turn,  can  be  better  correlated  to 
the  entries  on  the  survey. 

That  is,  without  pre-supposing  a  modular  design,  we  can  consider  that 
the  requirements  themselves  can  be  "modularized"  so  that  each  "module" 
can  be  evaluated  on  its  own  merit. 

Your  responses  on  sheets  1,  2  and  3  should,  therefore,  be  oriented 
towards  these  "modules"  rather  than  the  overall  computer  program. 


*  * 
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INSTRUCTIONS 


SHEET  1 


Sheet  1  itemizes  certain  INHERENT  factors  which  exist  in  a  software  product 
when  it  is  originally  conceived  and  defined.  They  are  the  operational  require¬ 
ments  which  the  system  is  expected  to  perform.  It  is  felt  that  the  require¬ 
ments  for  certain  capabilities  influence  the  complexity  and  "error-proneness" 
of  the  product  independent  of  the  development  methodologies  used.  In  reality, 
these  factors  influence  the  development  and  testing  methodologies. 

Column  entries  on  Sheet  1  are  relative  rankings  of  importance.  In  the  first 
column,  you  are  asked  to  rank  the  five  major  categories  against  one  another. 

In  the  second  column,  you  are  asked  to  rank  the  subcategories  within  each 
group. 

Row  entries  on  Sheet  1  are  categories  and  subcategories  of  inherent  factors 
which  have  been  singled  out  for  purposes  of  this  survey. 


ROW  DEFINITIONS 


SHEET  1 


OPERATIONAL  APPLICATION  TYPE  —  All  responses  to  this  survey  should  be  orient¬ 
ed  toward  characteristics  of  individual  modules.  The  purpose  of  each  module 
can  usually  be  used  to  determine  its  PREDOMINANT  application  type.  For  exautv- 
ple  if  the  purpose  of  the  module  is  to  issue  commamds  to  hardware  components 
we  would  say  the  the  module  is  of  the  "predominantly  control"  type  even 
though  it  includes  computational  commands. 

CONTROL  —  The  action  of  initiating,  sequencing,  terminating  or  otherwise 
influencing  the  operation  of  system  components  external  to  the  software. 

REAL-TIME  —  The  processing  of  information  or  data  in  a  manner  sufficiently 
rapid  that  the  results  of  the  processing  are  available  in  time  to  influ¬ 
ence  the  process  being  monitored  or  controlled. 

INPUT/OUTPUT  —  The  process  of  accepting  and  delivering  data  to  and  from 
system  conponents  external  to  the  software.  For  purposes  of  this  survey 
input/output  should  be  limited  to  file  input  and  report  output  activities 
as  opposed  to  the  control  type  described  above  or  the  interactive  type 
described  below. 

INTERACTIVE  —  A  method  of  conversational  input/output  wherein  the  software 
produces  eui  output  which  invokes  a  responsive  input  or  receives  an  input 
which  requires  a  responsive  output. 

COMPUTATIONAL  —  The  process  wherein  internally  available  data  is  combined, 
rearreuiged  and/or  otherwise  mauiipulated  to  alter  its  state.  For  example 
a  module  whose  purpose  is  to  convert  measurements  from  one  dimension  to 
another  should  be  regarded  as  being  computational. 
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MISSION  VARIABILITY  —  In  most  large  scale  software  applications,  a  variety  of 
"missions"  or  modes  of  operation  are  supported.  For  example,  software  re¬ 
quirements  for  embedded  software  in  a  missile  system  may  involve  distinct 
modes  of  operation  such  as  "pre-flight",  "boost"  and  "ballistic"  activities. 
Some  modules  will  perform  the  same  activities  regardless  of  the  mission 
type,  while  others  will  have  distinctly  different  characteristics  depending 
on  the  mission  mode.  MANY  and  SEVERAL  operational  missions  are  relative 
terms  that  may  be  interpreted  at  the  discretion  of  the  reader. 


FUNCTIONAL  COMPLEXITY  -  In  order  to  meet  its  intended  purpose,  a  module  may  be 
required  to  perform  more  theui  one  specific  task.  The  purpose  of  these 
entries  on  the  survey  is  to  accommodate  the  fact  that  some  functions  are 
relatively  easy  to  design  and  code  whereas  others  can  require  extensive  and 
highly  conplex  logic.  The  adjectives  used  are  relative  and  may  be  inter¬ 
preted  by  the  reader. 


SYSTEM  INTERACTION  —  This  category  is  a  refinement  of  earlier  categories. 
Interface  requirements  are  as  previously  defined.  EXTENSIVE  and  MINIMAL  are 
relative  terms  that  may  be  interpreted  by  the  reader.  Remember  that  you  are 
evaluating  individual  functional  requirements,  not  the  overall  software. 


INPUT  DOMAIN  VARIABILITY  —  This  category  is  a  refinement  of  earlier  catego¬ 
ries.  Here,  the  interest  is  not  in  the  quantity  of  inputs  required,  but 
rather  the  domain  from  which  it  comes.  For  example,  a  function  which  re¬ 
quires  "yes"  or  "no"  answers  to  many  questions  would  have  a  "NARRCW  RANGE" 
of  values  (yes  or  no).  On  the  other  hand,  a  single  input  of  an  angle  meas¬ 
urement  might  have  a  domain  of  -180.0000  to  +180.0000  degrees.  This  one 
would  be  considered  to  have  a  "WIDE  RANGE"  of  inputs. 


ERROR-PRONE/E3^0R-FREE  —  Itiese  adjectives  are  used  to  distinguish  the  effects 
on  module  reliability  caused  by  the  SOURCE  of  data  inputs.  A  device  which 
contains  self-checking  features  to  insure  that  its  inputs  to  the  computer 
are  correct  would  be  considered  "error-free".  On  the  other  hand,  other 
input  devices,  such  as  hunan  operators,  may  be  considered  to  be  "error- 
prone".  Despite  the  high  degree  of  subjectivity  in  rating  this  category, 
you  are  asked  to  rate  the  effects  introduced  by  such  situations. 


10 


INSTRUCTIONS 


SHEET  2 


Column  entries  on  Sheet  2  represent  phases  of  the  life  cycle  wherein  errors 
are  likely  to  be  introduced. 

Row  entries  on  Sheet  2  are  the  same  INHERENT  FACTORS  that  are  used  on  Sheet  1. 
their  definitions  are  repeated  below. 

You  are  asked  to  indicate  the  relative  number  of  errors  that  are  introduced  in 
each  phase  due  to  each  characteristic.  Please  remember  that  we  are  analyzing 
functions  (modules),  not  the  overall  software  product.  DO  NOT  "read  in"  any 
particular  methodology. 


COLUMN  DEFINITIONS 


SHEET 


REQUIREMENTS  DEFINITION  PHASE  -  This  is  the  period  of  time  during  which  the 
requirements  for  the  software  product,  such  as  functional  and  performance 
characteristics  are  defined  and  documented. 


PRELIMINARY  DESIGN  PHASE  -  During  this  phase  the  software  architecture  is 
defined  as  a  result  of  analysis  of  the  requirements  and  consideration  of 
possible  design  alternatives.  Typical  activities  include  the  definition 
and  structuring  of  conputer  programs,  components  and  data,  definition  of 
interfaces  and  preparation  of  timing  and  sizing  estimates. 

DETAILED  DESIGN  PHASE  -  During  this  phase,  the  preliminary  design  is  refined 
and  expanded  to  contain  more  detailed  descriptions  of  the  processing  logic, 
data  structures  and  input/output  requirements.  The  level  of  detail  must  be 
sufficient  for  implementation. 

CODING  OR  IMPLEMENTATION  PHASE  -  The  software  product  is  created  and  debugged 
during  this  phase.  The  detailed  design  is  implemented  via  a  computer  "lan¬ 
guage"  which  may  range  from  pure  binary  coded  instructions  to  a  very  high 
order  procedural  language.  For  purposes  of  this  survey,  testing  of  indiv¬ 
idual  software  components  are  considered  to  be  included  in  this  phase. 
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ROW  DEFINITIONS 


SHEET 


OPERATIONAL  APPLICATIOl  TYPE  —  All  responses  to  this  survey  should  be  orient¬ 
ed  toward  characteristics  of  individual  modules.  The  purpose  of  each  module 
can  usually  be  used  to  determine  its  PREDOMINANT  application  type.  For  exam¬ 
ple  if  the  purpose  of  the  module  is  to  issue  commauids  to  hardware  components 
we  would  say  the  the  module  is  of  the  "predominantly  control"  type  even 
though  it  includes  conpjtational  commands. 


CONTROL  —  The  action  of  initiating,  sequencing,  terminating  or  otherwise 
influencing  the  operation  of  system  conponents  external  to  the  software. 


REAL-TIME  —  The  processing  of  information  or  data  in  a  mcinner  sufficiently 
rapid  that  the  results  of  the  processing  are  available  in  time  to  influ¬ 
ence  the  process  being  monitored  or  controlled. 


INPUT/OUTPUT  —  The  process  of  accepting  and  delivering  data  to  and  from 
system  con^nents  external  to  the  software.  For  purposes  of  this  survey 
input/output  should  be  limited  to  file  input  emd  report  output  activities 
as  opposed  to  the  control  type  described  above  or  the  interactive  type 
described  below. 


INTE3?ACTIVE  —  A  method  of  conversational  input/output  wherein  the  software 
produces  an  output  v^ich  invokes  a  responsive  input  or  receives  an  input 
which  requires  a  responsive  output. 


COMPUTATIONAL  —  The  process  wherein  internally  available  data  is  combined, 
rearranged  and/or  otherwise  manipulated  to  alter  its  state.  For  example 
a  nodule  \diose  purpose  is  to  convert  measurements  from  one  dimension  to 
auiother  should  be  regarded  as  being  computational. 


MISSION  VARIABILITY  —  In  most  large  scale  software  applications,  a  variety  of 
"missions"  or  modes  of  operation  are  supported.  For  example,  software  re¬ 
quirements  for  embedded  software  in  a  missile  system  may  involve  distinct 
modes  of  operation  such  as  "pre-flight",  "boost"  and  "ballistic"  activities. 
Some  modules  will  perform  the  same  activities  regardless  of  the  mission 
type,  while  others  will  have  distinctly  different  characteristics  depending 
on  the  mission  mode.  MANY  and  SEVERAL  operational  missions  are  relative 
terms  that  may  be  interpreted  at  the  discretion  of  the  reader. 


FUNCTIONAL  COMPLEXITY  -  In  order  to  meet  its  intended  purpose,  a  module  may  be 
required  to  perform  more  than  one  specific  task.  The  purpose  of  these 
entries  on  the  survey  is  to  accommodate  the  fact  that  some  functions  are 
relatively  easy  to  design  and  code  whereas  others  can  require  extensive  and 
highly  complex  logic.  The  adjectives  used  are  relative  and  may  be  inter¬ 
preted  by  the  reader. 
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SYSTEM  INTERACTION  —  This  category  is  a  refinement  of  earlier  categories. 
Interface  requirements  are  as  previously  defined.  EXTENSIVE  and  MINIMAL  are 
relative  terms  that  may  be  interpreted  by  the  reader.  Remember  that  you  are 
evaluating  individual  functional  requirements,  not  the  overall  software. 


INPUT  DOMAIN  VARIABILITY  —  This  category  is  a  refinement  of  earlier  catego¬ 
ries.  Here,  the  interest  is  not  in  the  quantity  of  inputs  required,  but 
rather  the  domain  from  v^ich  it  comes.  For  example,  a  function  which  re¬ 
quires  "yes"  or  "no"  answers  to  nany  questions  would  have  a  "NARROW  RANGE" 
of  values  (yes  or  no).  On  the  other  hand,  a  single  input  of  an  auigle  meas¬ 
urement  might  have  a  domain  of  -180.0000  to  +180.0000  degrees.  This  one 
would  be  considered  to  have  a  "WIDE  RAIC-'^'."  of  inputs. 


ERROR-PRONE/131ROR-FREE  —  These  adjectives  are  used  to  distinguish  the  effects 
on  module  reli2d3ility  caused  by  the  SOURCE  of  data  inputs.  A  device  vdiich 
contains  self-checking  features  to  insure  that  its  inputs  to  the  con^ter 
are  correct  would  be  considered  "error-free".  On  the  other  h2uid,  other 
input  devices,  such  as  humeui  operators,  may  be  considered  to  be  "error- 
prone".  Despite  the  high  degree  of  subjectivity  in  rating  this  category, 
you  are  asked  to  rate  the  effects  introduced  by  such  situations. 


INSTRUCTIONS  -  SHEET  I 


Column  entries  on  Sheet  3  are  general  error  categories  and  ate  defined  belov/. 

Row  entries  on  Sheet  3  are  the  same  as  those  on  Sheets  1  and  2  and  are  re¬ 
peated  below. 

You  are  asked  to  distribute  whatever  errors  may  occur  into  the  four  categories 
listed.  On  this  sheet,  the  quantity  of  errors  is  irrelevant.  The  tiuestion 
being  asked  is,  "Assuming  that  the  row  entry  causes  errors,  what  peicent  falls 
into  each  category?" 


COLUMN  DEFINITIONS  -  SHEET  3 


LOGICAL  ERRORS  -  This  category  includes  all  instances  where  a  particular 
function  is  missing,  incorrect  or  inadequate  due  to  inadequate  requirements 
definition,  design  errors  or  omissions  or  implementation  errors. 


INTERFACE  ERRORS  -  This  category  includes  all  instances  where  a  recjuiced 
function  is  not  implemented  properly  due  to  irtiproper  communication  between 
system  components.  All  possible  interfaces  are  included  in  the  grouping: 

-  Software/Software:  Includes  errors  which  occur  between  software  components 
of  the  system  such  as  when  one  program  unit  fails  to  call,  calls  in  the 
wrong  sequence,  or  otherwise  improperly  calls  another  program  unit.  Also 
included  are  all  errors  resulting  from  the  improper  sharing  or  passing  of 
data  and/or  control  variables  between  program  units. 

-  Software/Hardware:  Includes  all  errors  which  result  in  loss  of  data  or 
untimely  excheunge  of  data  between  system  hardware  and  embedded  software. 
Included  are  situations  where  buffers  become  saturated  or  computation 
cycles  exceed  their  timing  allocations.  Also  included  are  errors  caused 
by  improper  data  exchange  between  system  hardware  and  embedded  softwaie. 

-  Software/Human:  SEE  INPUT/OUTPUT  ERRORS 


INPUT/OUTPUT  ERRORS  -  This  category  includes  all  instances  where  a  recjtiired 
function  is  not  properly  accomplished  due  to  the  manner  in  which  input  ot 
output  is  implemented.  For  purposes  of  this  survey,  include  in  this  cate¬ 
gory  all  software/human  interfaces.  For  example,  on  inp\jt  the  software  nvay 
either  accept  improper  commands  or  reject  proper  ones.  On  out[nit,  the  soft 
ware  may  generate  erroneous  or  ambiguous  messages  to  the  r^jietator  .  ftince 
the  survey  is  oriented  toward  mission  critical,  embedded  software,  l/O  fy? 
t'/^een  the  hardware  and  other  software  components  is  considered  m  t  he 
"Interface  Error"  category. 
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CXDMPUTATIONAL  E3®ORS  -  These  ace  calculation  errors.  Included  are;  errors  of 
omission  such  as  uninitialized  variables;  mathematical  errors  such  as  incor 
rect  expressions,  conversion  and  truncation  errors;  and  programming  errors 
such  as  inproper  use  of  indices,  variables  and  overlays.  DO  NOT  INCLUDE 
SYNTAX  ERRORS  that  would  be  eliminated  prior  to  operation. 
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ROW  DEFINITIONS 


SHEET  3 


OPERATIONAL  APPLICATICX^  TYPE  —  All  responses  to  this  survey  should  be  orient¬ 
ed  toward  characteristics  of  individual  modules.  The  purpose  of  each  module 
C2U1  usually  be  used  to  determine  its  PREDOMINANT  application  type.  For  exam¬ 
ple  if  the  purpose  of  the  module  is  to  issue  commands  to  hardware  components 
we  would  say  that  the  module  is  of  the  "predominantly  control"  type  even 
though  it  includes  computational  commands. 

CONTROL  —  The  action  of  initiating,  sequencing,  terminating  or  otherwise 
influencing  the  operation  of  system  components  external  to  the  software. 

REAL-TIME  —  ihe  processing  of  information  or  data  in  a  manner  sufficiently 
rapid  that  the  results  of  the  processing  are  available  in  time  to  influ¬ 
ence  the  process  being  monitored  or  controlled. 

INPUT/OUTPUT  —  The  process  of  accepting  and  delivering  data  to  and  from 
system  components  external  to  the  software.  For  purposes  of  this  survey 
input/output  should  be  limited  to  file  input  and  report  output  activities 
as  opposed  to  the  control  type  described  eibove  or  the  interactive  type 
described  below. 

INTERACTIVE  —  A  method  of  conversational  input/output  wherein  the  software 
produces  an  output  which  invokes  a  responsive  input  or  receives  an  input 
which  requires  a  responsive  output. 

COMPUTATIONAL  —  The  process  v^erein  internally  available  data  is  combined, 
rearranged  and/or  otherwise  manipulated  to  alter  its  state.  For  example 
a  module  whose  purpose  is  to  convert  measurements  from  one  dimension  to 
another  should  be  regarded  as  being  computational. 


MISSION  VARIABILITY  —  In  most  large  scale  software  applications,  a  variety  of 
"missions"  or  modes  of  operation  are  supported.  For  example,  software  re¬ 
quirements  for  embedded  software  in  a  missile  system  may  involve  distinct 
inodes  of  operation  such  as  "pre-flight",  "boost"  euid  "ballistic"  activities. 
Some  modules  will  perform  the  same  activities  regardless  of  the  mission 
type,  while  others  will  have  distinctly  different  characteristics  depending 
on  the  mission  mode.  MANY  eind  SEVERAL  operational  missions  are  relative 
terms  that  may  be  interpreted  at  the  discretion  of  the  reader. 


FUNCnCWAL  COMPLEXITY  -  In  order  to  meet  its  intended  purpose,  a  module  may  be 
required  to  perform  more  them  one  specific  task.  The  purpose  of  these 
entries  on  the  survey  is  to  accommodate  the  fact  that  some  functions  are 
relatively  easy  to  design  and  code  whereas  others  can  require  extensive  and 
highly  complex  logic.  The  adjectives  used  are  relative  and  may  be  inter¬ 
preted  by  the  reader. 
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SYSTEM  INTERACTION  —  This  category  is  a  refinement  of  earlier  categories. 
Interface  requirements  are  as  previously  defined.  EXTENSIVE  eund  MINIMAL  are 
relative  terms  that  may  be  interpreted  by  the  reader.  Remember  that  you  are 
evaluating  individual  functional  requirements,  not  the  overall  software. 


INPUT  DOMAIN  VARIABILITY  —  This  category  is  a  refinement  of  earlier  catego¬ 
ries.  Here,  the  interest  is  not  in  the  qucintity  of  inputs  required,  but 
rather  the  domain  from  vdiich  it  comes.  For  example,  a  function  which  re¬ 
quires  "yes"  or  "no"  answers  to  many  questions  would  have  a  "NARROW  RANGE" 
of  values  (yes  or  no).  On  the  other  hand,  a  single  input  of  an  angle  meas¬ 
urement  might  have  a  domain  of  -180.0000  to  +180.0000  degrees.  This  one 
would  be  considered  to  have  a  "WIDE  RANGE"  of  inputs. 


ERROR-PRONE/ERROR-FREE  —  These  adjectives  are  used  to  distinguish  the  effects 
on  module  reliaJaility  caused  by  the  SOURCE  of  data  inputs.  A  device  vtiich 
contains  self-checking  features  to  insure  that  its  inputs  to  the  computer 
are  correct  would  be  considered  "error-free".  On  the  other  heuid,  other 
input  devices,  such  as  human  operators,  may  be  considered  to  be  "error- 
prone".  Despite  the  high  degree  of  subjectivity  in  rating  this  category, 
you  are  asked  to  rate  the  effects  introduced  by  such  situations. 
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INSTRUCTIONS  -  SHEET  4 


Column  entries  on  Sheet  4  are  the  same  general  error  categories  used  on  Sheet 
3  and  their  definitions  are  repeated  below. 

Row  entries  on  Sheet  4  "error  AVOIDANCE  mechanisms"  which  are  effective  in 
minimizing  or  reducing  errors. 

You  are  asked  to  evaluate  the  effectiveness  of  each  mechanism  by  indicating 
what  percentage  of  each  error  type  caui  be  avoided  by  the  use  of  that  mech¬ 
anism. 


COLUMN  DEFINITIONS  -  SHEET  4 


LOGICAL  EI^ORS  -  This  category  includes  all  instances  where  a  particular 
function  is  missing,  incorrect  or  inadequate  due  to  inadequate  requirements 
definition,  design  errors  or  omissions  or  implementation  errors. 

INTERFACE  ERRORS  -  This  category  includes  all  instances  where  a  required 
function  is  not  implemented  properly  due  to  improper  communication  between 
system  components.  All  possible  interfaces  are  included  in  the  grouping: 

-  Software/Software:  Includes  errors  which  occur  between  software  components 
of  the  system  such  as  when  one  program  unit  fails  to  call,  calls  in  the 
wrong  sequence,  or  otherwise  iitproperly  calls  another  program  unit.  Also 
included  are  all  errors  resulting  from  the  improper  sharing  or  passing  of 
data  and/or  control  variables  between  program  units. 

-  Software/Hardware:  Includes  all  errors  which  result  in  loss  of  data  or 
untimely  exchemge  of  data  between  system  hardware  and  embedded  software. 
Included  are  situations  where  buffers  become  saturated  or  computation 
cycles  exceed  their  timing  allocations.  Also  included  are  errors  caused 
by  improper  data  exchange  between  system  hardware  and  embedded  software. 

-  Software/Human:  SEE  INPUT/OUTPUT  ERRORS 


INPUT/OUTPUT  ERRORS  -  This  category  includes  all  instances  where  a  required 
function  is  not  properly  accomplished  due  to  the  manner  in  which  input  or 
output  is  implemented.  For  purposes  of  this  survey,  include  in  this  cate¬ 
gory  all  software/human  interfaces.  For  example,  on  input  the  software  may 
either  accept  inproper  commands  or  reject  proper  ones.  On  output,  the  soft¬ 
ware  may  generate  erroneous  or  ambiguous  messages  to  the  operator.  Since 
the  survey  is  oriented  toward  mission  critical,  embedded  software,  I/O  be¬ 
tween  the  hardware  and  other  software  components  is  considered  in  the 
"Interface  Error"  category. 
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COMPUTATIONAL  E3«lORS  -  These  are  calculation  errors.  Included  are:  errors  of 
omission  such  as  uninitialized  variables;  mathematical  errors  such  as  incor¬ 
rect  expressions,  conversion  euid  truncation  errors;  amd  prograuiming  errors 
such  as  improper  use  of  indices,  variables  and  overlays.  DO  NOT  INCLUDE 
SYNTAX  ERRORS  that  would  be  eliminated  prior  to  operation. 
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ROW  DEFINITIONS  -  SHEET  4 


QUALITY  ASSURANCE  ORGANIZATION  —  A  group  responsible  for  the  planned  and 
systematic  review  of  the  software  development  process  and  its  products  to 
provide  adequate  confidence  that  the  item  or  product  conforms  to  established 
technical  requirements. 

TEST  ORGANIZATION  —  A  group  responsible  for  preparing  test  pleuis  and  proce¬ 
dures,  executing  the  test  procedures,  and  analyzing  the  test  results  in 
order  to  verify  that  the  system  performed  its  intended  functions.  This 
group  is  also  responsible  for  documenting  problems  detected  during  testing 
and  verifying  by  retest  that  corrections  to  the  software  work  properly, 

INDEPENDENT  VERIFICATICW  AND  VALIDATION  (IV&V)  —  Verification  and  validation 
of  a  software  product  performed  by  eun  organization  that  is  both  technically 
eind  managerially  separate  from  the  organization  responsible  for  developing 
the  product. 

SOFTWARE  SUPPORT  LIBRARY  —  A  software  library  containing  computer  readable 
and  humaun  readable  information  relevant  to  a  software  developtvent  effort. 

CONFIGURATION  CCWTROL  BOARD  —  The  authority  responsible  for  evaluating  and 
approving  or  disapproving  proposed  engineering  changes,  and  ensuring 
implementation  of  the  approved  changes. 

SOFTWARE  DEVELOPMENT  PLAN  —  This  document  presents  the  comprehensive  plan 
for  the  project's  software  development  activities  by  describing  the  software 
development  organization,  the  software  design  and  testing  approach,  the  pro¬ 
grams  euid  documentation  that  will  be  produced,  software  milestones  and 
schedules,  euid  the  allocation  of  development  resources. 

SYSTEM  REQUlr<EMENTS  SPECIFICATION  —  This  document  states  the  technical  and 
mission  requirements  for  a  system  as  ain  entity,  allocates  requirements  to 
functional  areas,  and  defines  the  interfaces  between  or  among  the  functional 
areas. 

INTERFACE  DESIGN  SPECIFICATION  —  This  is  an  optional  document  which  is  re¬ 
quired  whenever  the  system  contains  two  or  more  computers  that  must  com¬ 
municate  with  each  other.  It  provides  a  detailed  logical  description  of  all 
data  units,  messages,  control  signals  and  communication  conventions  between 
the  digital  processors. 

SOFTWARE  REQUIREMENTS  SPECIFICATION  —  This  document  establishes  the  require¬ 
ments  for  the  performance,  design,  test  and  qualification  of  the  computer 
program. 

SOFTWARE  FUNCTIONAL  DESIGN  SPECIFICATION  —  This  document  establishes  the 
functional  design  of  the  software  at  the  computer  program  level.  It  provides 
sufficient  design  information  to  accomplish  the  goals  of  the  Preliminary 
Design  Review. 
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SOFTOARE  DETAILED  DESIGN  SPECIFICATION  ~  This  doCTJment  provides  conplete 
programming  design  sufficiently  detailed  for  a  programmer  to  code  from  with 
minimal  additional  direction. 

REQUIREMENTS  TRACEABILITY  MATRIX  —  A  set  of  tables  which  provides  traceabil¬ 
ity  of  software  requirements  from  the  system  specification  to  the  individual 
item  requirements  specifications,  to  the  design  specification  which  imple¬ 
ments  the  requirements,  and  to  the  software  plauis  and  procedures  that  verify 
that  requirements  have  been  fully  implemented. 

STRUCTURED  ANALYSIS  TOOLS  —  These  define  a  systematic  method  of  using  func¬ 
tion  networks  aind  other  tools  to  develop  aui  analysis-phase  model  of  a  sys¬ 
tem.  Typical  tools  include  Data  Flow  Diagrams,  Data  Dictionaries  and  struc¬ 
tured  English. 

PROGRAM  SPECIFICATICW  LANGUAGE  (PSL)  ~  A  language  used  to  specify  the  re¬ 
quirements,  design,  behavior,  or  other  characteristics  of  a  system  or  sys¬ 
tem  conponent. 

PROGRAM  DESIGN  LANGUAGE  (PDL)  —  A  language  with  special  constructs  and,  some¬ 
times,  verification  protocols  used  to  develop,  analyze,  and  document  a  de¬ 
sign. 

HIGH  ORDER  LANGUAGE  (HOL)  —  A  programming  language  v^ich  provides  compression 
of  computer  instructions  such  that  one  program  statement  represents  many 
machine  language  instructions.  It  is  non-problem  specific  and  is  used  by 
programmers  to  communicate  with  the  conputer. 

HIEDRARCHICAL  DESIGN  —  A  design  which  consists  of  multiple  levels  of  decom¬ 
position,  general  to  specific.  It  is  a  structured  approach  with  the  addi¬ 
tional  restriction  that  program  control  is  accomplished  hiearachially.  That 
is,  program  modules  may  only  invoke  other  modules  which  are  subordinate  to 
them. 

TOP-DCWN  DESIGN  —  An  ordering  to  the  sequence  of  decisions  which  are  made  in 
the  decoirposition  of  a  software  system,  by  beginning  with  a  simple  descrip¬ 
tion  of  the  entire  process  (top  level).  Through  a  succession  of  refinements 
of  what  has  been  defined  at  each  level,  lower  levels  are  specified. 

STRUCTURED  DESIGN  —  A  disciplined  approach  to  software  design  which  adheres 
to  a  specified  set  of  rules  based  on  principles  such  as  top-down  design, 
modularization,  stepwise  refinement,  etc. 

SINGLE  FUNCTION  MODULARIZATION  —  An  organization  of  the  functions  of  the 
computer  program  into  a  set  of  discrete  program  modules  each  of  which  is 
designed  to  perform  a  single  function. 

STRUCTURED  CODE  —  Code  that  has  been  generated  with  a  limited  number  of  well- 
defined  control  structures  using  stepwise  refinement. 


AUTOMATIC  MEASUREMENT  TOOLS  —  This  category  includes  all  computer  programs 
which  evaluate  other  computer  programs.  They  may  be  used  to  verify  compli- 
auice  with  coding  standards,  to  measure  progress,  or  to  provide  a  measure  of 
complexity.  They  may  be  applied  to  any  or  all  phases  of  the  development 
cycle. 

AUTOMATIC  TEST  TOOLS  —  This  category  includes  all  computer  programs  that 
automatically  devise  and/or  execute  tests  on  other  computer  programs  by 
einalysis  of  the  path  logic  and  variable  domains  of  the  software  being  tested 
cind  construction  of  test  data  sets  vdiich  will  exercise  all  logical  paths 
under  all  or  extreme  input  conditions. 
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INSTRUCTIONS  -  SHEET  5 


Column  entries  on  Sheet  5  are  the  same  general  error  categories  used  on  Sheets 
3  and  4  and  their  definitions  are  defined  below. 

Row  entries  on  Sheet  5  are  "error  DETECTICW  mechanisms"  vdiich  are  effective  in 
detecting  or  recognizing  errors. 

You  are  asked  to  evaluate  the  effectiveness  of  each  mechanism  by  indicating 
what  percentage  of  each  error  type  can  be  detected  (and  corrected)  by  the  use 
of  that  mechcinism.  Ignore  the  possibility  that  the  mechanism  itself  may 
introduce  errors. 


COLUMN  DEFINITIONS  -  SHEET  5 


LOGICAL  ERRORS  -  This  category  includes  all  instauices  where  a  particular 
function  is  missing,  incorrect  or  inadequate  due  to  inadequate  requirements 
definition,  design  errors  or  omissions  or  implementation  errors. 


INTERFACE  ERRORS  -  This  category  includes  all  instances  where  a  required 
function  is  not  implemented  properly  due  to  improper  communication  between 
system  components.  All  possible  interfaces  are  included  in  the  grouping: 

-  Software/Software:  Includes  errors  vdiich  occur  between  software  components 
of  the  system  such  as  vdien  one  program  unit  fails  to  call,  calls  in  the 
wrong  sequence,  or  otherwise  iitproperly  calls  emother  program  unit.  Also 
included  are  all  errors  resulting  from  the  improper  sharing  or  passing  of 
data  eind/or  control  variables  between  program  units. 

-  Software/Hardware:  Includes  all  errors  which  result  in  loss  of  data  or 
untimely  exchange  of  data  between  system  hardware  and  embedded  software. 
Included  are  situations  where  buffers  become  saturated  or  computation 
cycles  exceed  their  timing  allocations.  Also  included  are  errors  caused 
by  improper  data  exchange  between  system  hardware  and  embedded  software. 

-  Software/Human:  SEE  INPUT/CXJTPUT  ERRORS 


INPUT/OUTPUT  ERRORS  -  This  category  includes  all  instemces  where  a  required 
function  is  not  properly  accomplished  due  to  the  nanner  in  which  input  or 
output  is  implemented.  For  purposes  of  this  survey,  include  in  this  cate¬ 
gory  all  software/hunan  interfaces.  For  example,  on  input  the  software  may 
either  accept  improper  commands  or  reject  proper  ones.  On  output,  the  soft¬ 
ware  may  generate  erroneous  or  ambiquous  messages  to  the  operator.  Since 
the  survey  is  oriented  toward  mission  critical,  embedded  software,  I/O  be¬ 
tween  the  hardware  emd  other  software  components  is  considered  in  the 
"Interface  Error"  category. 
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COMPUTATIONAL  ERRORS  -  These  are  calculation  errors.  Included  are:  errors  of 
omission  such  as  uninitialized  variables;  mathematical  errors  such  as  incor 
rect  expressions,  conversion  auid  truncation  errors;  and  programming  errors 
such  as  improper  use  of  indices,  variables  and  overlays.  DO  NOT  INCLUDE 
SYNTAX  ERRORS  that  would  be  eliminated  prior  to  operation. 
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ROW  DEFINITIONS 


SHEET  5 


FREQUENT/INFREQUENT  —  These  are  relative  terms  that  may  be  interpreted  by  the 
reader.  In  general,  however,  it  is  preferred  that  "frequent"  be  used  to 
describe  activities  that  occur  on  a  regular,  scheduled  basis  (e.g.,  weekly). 
"Infrequent"  carries  the  connotation  that  the  activity  is  less  rigidly 
planned  and  accomplished  (e.g.,  whenever  a  problem  is  suspected). 

WALKTHROUGH  —  A  review  process  in  v^ich  an  analyst,  designer  or  programmer 
leads  one  or  more  peers  through  a  segment  of  the  software  product  which  he 
or  she  has  developed. 

PROGRESS  REVIEW  —  For  purposes  of  this  survey,  a  progress  review  is  a  peri¬ 
odic  report  given  to  an  individual's  supervisor  to  provide  an  assessment  of 
the  state  of  conpletion  of  a  software  product.  This  is  in  contrast  to  a 
walkthrough  which  is  conducted  among  peers  and  is  primarily  technical  in 
nature. 

QUALITY  AUDIT  —  For  purposes  of  this  survey,  a  quality  audit  is  eui  euinounced 
or  unannounced  inspection  of  a  software  product  or  process.  For  exeunple,  an 
audit  may  consist  of  am  inspection  of  a  portion  of  a  prograimmer' s  code  to 
verify  conpliance  with  programming  standards. 

PQT  and  FQT  —  See  Below 

SOFTWARE  PROBLEM  REPORT  —  A  report  of  a  program  deficiency  identified  during 
software  qualification,  test,  system  integration  and  test,  or  system  oper¬ 
ation,  which  appears  to  be  software  related. 

SPECIFICATION  CHANGE  NOTICE  —  A  formal  notification  of  a  change  in  the 
specification. 

ENGINEERING  CHANGE  NOTICE  —  A  document  used  to  process  changes  to  baseline 
documents  and  which  includes  both  notice  of  an  engineering  change  to  a 
configuration  item  and  the  supporting  documentation  by  which  the  chemge  is 
described. 

SOFTWARE  REQUIREMENTS  REVIEW  (SRR)  —  A  review  to  achieve  formal  agreement 
between  the  customer  and  the  developer  that  the  software  requirements 
specifications  are  complete  and  accurate. 

PRELIMINARY  DESIGN  REVIEW  (PDR)  —  A  formal  technical  review  of  the  basic 
design  approach.  It  is  held  after  the  conpletion  of  preliminary  design 
efforts  but  prior  to  the  start  of  detailed  design.  See  also  SYSTEM 
DESIGN  REVIEW  and  CRITICAL  DESIGN  REVIEW. 

CRITICAL  DESIGN  REVIEW  (CDR)  —  A  formal  technical  design  review  conducted 
to  ensure  that  the  detailed  design  correctly  and  completely  satisfies  the 
requirements.  It  is  conducted  after  completion  of  the  detailed  design  but 
prior  to  coding.  It  estedilishes  the  design  baseline. 
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TEST  READINESS  REVIEW  (TRR)  —  A  review  conducted  prior  to  each  test  to  ensure 
adequacy  of  the  documentation  and  to  establish  a  configuration  baseline. 

FUNCTIONAL  CONFIGURATION  AUDIT  (FCA)  --  Audit  to  verify  that  the  actual  per- 
formeuice  of  the  configuration  items  cortplies  with  the  B-5  develojxnent 
specifications. 

PHYSICAL  CONFIGURATION  AUDIT  (PCA)  —  A  formal  examination  of  the  as-built 
version  of  a  configuration  item  against  its  technical  documentation  to 
ensure  the  adequacy,  conpleteness,  and  accuracy  of  the  technical  design 
documentation . 

UNIT  LEVEL  TESTING  —  Testing  to  verify  program  unit  logic,  coitputational 
adequacy,  data  handling  capability,  interfaces  and  design  extremes,  and  to 
execute  and  verify  every  branch. 

PRELIMINARY  QUALIFICATION  TESTING  (PQT)  —  An  incremental  testing  process 
which  provides  visibility  and  control  of  the  computer  program  development 
during  the  time  period  between  the  Critical  Design  Review  (CDR)  and  Formal 
Qualification  Testing  (FQT);  conducted  for  those  functions  critical  to  the 
CPCI. 

FORMAL  QUALIFICATION  TESTING  (FQT)  —  Testing  conducted  prior  to  Functional 
Configuration  Audit  to  demonstrate  CPCI  conpliemce  with  all  applicable 
software  specifications. 

SOFTWARE  INTEGRATION  TESTING  —  Tests  of  the  overall  computer  program  used  to 
verify  proper  module  interfaces  with  respect  to  sequencing,  timing,  and  data 
compatibility. 

SYSTEM  INTEGRATION  TESTING  —  The  process  of  testing  an  integrated  hardware 
2uid  software  system  to  verify  that  the  system  meets  its  specified  require¬ 
ments. 

OPERATIONAL  FIELD  TESTING  —  Tests  performed  by  operational  personnel  in  the 
operational  environment.  These  can  be  the  same  tests  performed  earlier 
during  FQT. 
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FULL  SCALE  SURVEY  RESULTS 


Three  hundred  and  fifty  surveys  were  distributed  euid  ninety  were  returned. 
Table  F-1  shows  a  profile  of  the  qualifications  of  the  people  who  responded  to 
the  survey.  Itie  occupations  identified,  the  education  level,  and  the  years 
experience  show  that  we  were  quite  successful  in  tapping  genuine  expert  opin¬ 
ions.  The  results  presented  herein  represent  the  statistical  properties  of 
the  entire  sample.  Several  test  cases  were  conducted  to  see  if  there  was  a 
noticeable  difference  in  the  responses  of  particular  groups  of  individuals 
(e.g.,  government  versus  industry),  but  none  was  apparent. 

Table  F-2  shows  the  results  of  sheet  1  of  the  survey  where  participants  were 
asked  to  rauik  the  importauice  of  various  factors  auid  categories.  No  single  cat¬ 
egory  was  universally  recognized  as  the  most  (or  least)  important.  The  re¬ 
sults,  however,  are  indicative  of  the  wide  spectrum  of  software  applications 
and  the  subsequent  divergence  of  concerns  among  developers.  That  is,  the 
primary  concerns  of  software  developers  eutiu  users  are  directly  related  to  the 
type  of  software  with  which  they  are  associated.  It  also  indicates  that  a 
reliability  prediction  methodology  must  either  be  "customized"  for  each  appli¬ 
cation  or  must  include  provisions  for  all  possible  influences. 

Table  F-3  presents  the  statistical  results  of  the  survey.  Specifically,  the 
following  statistics  are  shown: 

o  AVERAGE  -  This  is  the  numerical  average  of  all  the  responses  received. 

It  was  computed  individually  for  each  row-column  entry  by  summing  the 
non-blank  entries  2Uid  dividing  by  the  number  of  non-blauik  entries. 

o  BOUNDS  -  These  define  the  90%  confidence  interval  based  on  the  computer 
st2uidard  deviation  of  the  mean  response. 

o  STANDARD  DEVIATION  -  This  is  the  Standard  deviation  of  the  mean,  based 
on  the  computed  sample  standard  deviation  and  the  sample  size  of  each 
individual  row-column  entry. 

o  NUMBER  OF  RESPONSES  -  This  is  the  number  of  non-bleuik  responses  to 
each  individual  entry. 


TABLE  F-2  RANKING  OF  INHERENT  CHARACTERISTICS 


OPERATIONAL  APPLICATION  TYPE 

Predomineuitly  Control 
Predominantly  Real  Time 
Predominantly  Input/Output 
Predominantly  Interactive 
Predominantly  Computational 


MISSION  VARIABILITY 

Many  Distinct  Operational  Missions 
Several  Variations  of  Operational  Missions 
Single  Operational  Mission 


FUNCTIONAL  COMPLEXITY 


Many  Operations  Required 
Many  Operations  Required 
Few  Operations  Required 
Few  Operations  Required 


Highly  Complex 
Relatively  Simple 
Highly  Conplex 
Relatively  Sirtple 


SYSTEM  INTERACTION 

Elxtensive  Hardware  Interface  Requirements 
Minimal  Hardware  Interface  Requirements 
Extensive  Software  Interface  Requirements 
Minimal  Software  Interface  Requirements 
Extensive  Hurwn  Interface  Requirements 
Minimal  Human  Interface  Requirements 


INPUT  DOMAIN  VARIABILITY 

Wide  Reuige  of  Error-Prone  Inputs 
Wide  Range  of  Error-Free  Inputs 
Narrow  Range  of  Error-Prone  Inputs 
Narrow  Range  of  Error-Free  Inputs 
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■TABLE  F-3.  STATISTICAL  RESULTS  OF  SURVEY 


INHERENT  FACTORS 


PREDOMINANTLY  CONTROL 


PREDOMINANTLY  REAL  TIME 


PREDOMINANTLY  INPUT/OUTPUT 


REQ 

PRELIM 

DETAIL 

DEF 

DESIGN 

DESIGN 

CODE 

Average 

5.9 

4.7 

4.7 

3.1 

Upper  Bound 

6.5 

5.2 

5.2 

3.6 

Lower  Bound 

5.3 

4.2 

4.3 

2.6 

Standard  Deviation 

0.4 

0.3 

0.3 

0.3 

Number  of  Responses 

79 

80 

79 

78 

Average 

6.1 

5.8 

6.3 

4.4 

Upper  Bound 

6.7 

6.4 

6.7 

5.0 

Lower  Bound 

5.5 

5.3 

5.8 

3.9 

Standard  Deviation 

0.3 

0.3 

0.3 

0.4 

Number  of  Responses 

82 

82 

80 

79 

r 

Average 

5.3 

4.4 

4.3 

3.0 

Upper  Bound 

5.9 

4.9 

4.7 

3.5 

Lower  Bound 

4.7 

3.9 

3.9 

2.5 

Standard  Deviation 

0.4 

0.3 

0.3 

0.3 

Number  of  Responses 

80 

80 

80 

76 

PREDOMINANTLY  INTERACTIVE 


Average 

6.2 

5.1 

4.8 

3.6 

Upper  Bound 

6.8 

5.6 

5.3 

4.1 

Lower  Bound 

5.6 

4.5 

4.3 

3.0 

Standard  Deviation 

0.4 

0.3 

0.3 

0.3 

Number  of  Responses 

78 

78 

78 

78 

PREDOMINANTLY  COMPUTATIONAL 


Average 

3.7 

3.3 

3.5 

3.0 

7T77 

r-  V  ■ 

Upper  Bound 

4.2 

3.8 

3.9 

3.6 

«-  n  • 

Lower  Bound 

3.1 

2.8 

3.0 

2.5 

Standard  Deviation 

0.3 

0.3 

0.3 

0.3 

Number  of  Responses 

79 

78 

80 

80 

F  -  5 


REQ  PRELIM  DETAIL 

INHERENT  FACTORS  DEF  DESIGN  DESIGN 


MANY  DISTINCT  OPERATIONAL  MISSIONS 


Average 

7.4 

6.4 

5.6 

Upper  Bound 

7.9 

6.9 

6.1 

Lower  Bound 

7.0 

5.9 

5.1 

Stemdard  Deviation 

0.3 

0.3 

0.3 

Number  of  Responses 

82 

79 

78 

SEVERAL  VARIATICNS  OF  OPERATIONAL  MISSIONS 

Average 

6.2 

5.4 

4.5 

Upper  Bound 

6.7 

5.8 

5.0 

Lower  Bound 

5.6 

4.9 

4.1 

Standard  Deviation 

0.3 

0.3 

0.3 

Number  of  Responses 

82 

79 

77 

SINGLE  OPERATIONAL  MISSION 

Average 

3.3 

2.8 

2.5 

Upper  Bound 

3.9 

3.2 

2.9 

Lower  Bound 

2.8 

2.4 

2.1 

Standard  Deviation 

0.3 

0.3 

0.3 

Number  of  Responses 

82 

78 

78 

MANY  OPERATIONS  REQUIRED  -  HIGHLY  COMPLEX 

Average 

7.7 

7.1 

6.5 

Upper  Bound 

8.2 

7.5 

6.9 

Lower  Bound 

7.2 

6.7 

6.0 

Standard  Deviation 

0.3 

0.2 

0.3 

Number  of  Responses 

86 

84 

84 

MANY  OPERATIONS  REQUIRED  -  RELATIVELY  SIMPLE 

Average 

5.1 

4.6 

3.8 

Upper  Bound 

5.6 

5.0 

4.2 

Lower  Bound 

4.5 

4.1 

3.3 

Standard  Deviation 

0.3 

0.3 

0.3 

Number  of  Responses 

80 

81 

80 

INHERENT  FACTORS 

REQ 

DEF 

PRELIM 

DESIGN 

DETAIL 

DESIGN 

CODE 

FEW  OPERATIONS  REQUIRED 

-  HIGHLY  COMPLEX 

Average 

6.0 

5.6 

5.2 

3.2 

Upper  Bound 

6.6 

6.1 

5.7 

3.7 

Lower  Bound 

5.5 

5.1 

4.8 

2.7 

Standard  Deviation 

0.3 

0.3 

0.3 

0.3 

Number  of  Responses 

83 

81 

81 

79 

FEW  OPERATIONS  REQUIRED  -  RELATIVELY  SIMPLE 


Average 

2.9 

2.5 

2.2 

1.8 

Upper  Bound 

3.3 

2.9 

2.6 

2.1 

Lower  Bound 

2.4 

2.1 

1.8 

1.4 

Standard  Deviation 

0.3 

0.2 

0.2 

0.2 

Number  of  Responses 

79 

81 

82 

80 

EXTENSIVE  HARDWARE  INTERFACE  REQUIREMENTS 

Average 

6.9 

6.3 

5.9 

4.2 

Upper  Bound 

7.5 

6.8 

6.3 

4.8 

Lower  Bound 

6.3 

5.9 

5.5 

3.6 

Standard  Deviation 

0.3 

0.3 

0.3 

0.4 

Number  of  Responses 

80 

80 

79 

74 

MINIMAL  HARDWARE  INTERFACE  REQUIREMENTS 

Average 

3.6 

3.0 

2.5 

1.9 

Uj^r  Bound 

4.1 

3.5 

2.9 

2.3 

Lower  Bound 

3.1 

2.6 

2.0 

1.5 

Stendard  Deviation 

0.3 

0.3 

0.3 

0.2 

Number  of  Responses 

74 

75 

• 

76 

73 

EXTENSIVE  SOFTWARE  INTERFACE  REQUIREMENTS 

Average 

7.0 

7.0 

6.5 

4.5 

Up^r  Bound 

7.5 

7.4 

6.9 

5.1 

Lower  Bound 

6.5 

6.5 

6.1 

3.9 

Standard  Deviation 

0.3 

0.3 

0.2 

0.4 

Number  of  Responses 

82 

81 

81 

77 

F  -  7 


INHERENT  FACTORS 


MINIMAL  SOFTWARE  INTERFACE  REQUIREMENTS 


EXTENSIVE  HUMAN  INTERFACE  REQUIREMENTS 


MINIMAL  HUMAN  INTERFACE  REQUIREMENTS 


WIDE  RANGE  OF  ERROR-PRONE  INPUTS 


WIDE  RANGE  OF  ERROR-FREE  INPUTS 


PRELIM 

DESIGN 


DETAIL 

DESIGN 


Average 

3.3 

2.9 

2.4 

Upper  Bound 

3.8 

3.3 

2.8 

Lower  Bound 

2.8 

2.5 

2.0 

Staundard  Deviation 

0.3 

0.3 

0.3 

Number  of  Responses 

75 

77 

77 

Average 

7.1 

6.7 

6.0 

Upper  Bound 

7.7 

7.2 

6.4 

Lower  Bound 

6.6 

6.2 

5.6 

Standard  Deviation 

0.3 

0.3 

0.3 

Number  of  Responses 

82 

79 

80 

Average 

3.4 

2.9 

2.6 

Upper  Bound 

3.9 

3.4 

3.1 

LcTwer  Sound 

2.9 

2.4 

2.2 

Stauidard  Deviation 

0.3 

0.3 

0.3 

Number  of  Responses 

75 

76 

77 

Average 

6.5 

6.4 

6.1 

Upper  Bound 

7.1 

6.9 

6.6 

Lower  Bound 

5.9 

5.9 

5.6 

Stauidard  Deviation 

0.3 

0.3 

0.3 

Number  of  Responses 

79 

79 

80 

Average 

4.2 

4.1 

3.4 

Upper  Bound 

4.7 

4.6 

3.9 

Lower  Bound 

3.7 

3.6 

2.9 

Stauidard  Deviation 

0.3 

0.3 

0.3 

Number  of  Responses 

78 

77 

75 

F  -  8 


INHERENT  FACTORS 


NARROW  RANGE  OF  ERROR-PRONE  INPUTS 


NARROW  RANGE  OF  ERROR-FREE  INPUTS 


F  -  9 


V,  X  V-'*  .  •  •  •  •  •  • 


PRELIM  DETAIL 
DESIGN  DESIGN 


'  'fc*  '  ^  ^  ■  J*’  '■ 


Average 

4.8 

4.6 

3.9 

Upper  Bound 

5.4 

5.1 

4.5 

Lower  Bound 

4.3 

4.2 

3.4 

Standard  Deviation 

0.3 

0.3 

0.3 

Number  of  Responses 

76 

76 

75 

Average 

2.7 

2.4 

2.1 

Upper  Bound 

3.2 

2.9 

2.5 

Lower  Bound 

2.2 

2.0 

1.7 

Standard  Deviation 

0.3 

0.3 

0.2 

Number  of  Responses 

76 

75 

76 

v,-v 


INHERENT  FACTORS 

LOGIC 

INT 

I/O 

COMP 

PREDOMINANTLY  CONTROL 

Average 

37.2 

29.9 

19.2 

13.8 

Upper  Boiind 

40.9 

32.8 

21.4 

15.8 

Lower  Bound 

33.5 

27.0 

16.9 

11.8 

Standard  Deviation 

2.3 

1.8 

1.4 

1.2 

Number  of  Responses 

75 

75 

75 

75 

PREDOMINANTLY  REAL  TIME 

Average 

30.8 

30.8 

20.3 

16.6 

Upper  Bound 

34.1 

33.6 

22.9 

19.4 

Lower  Bound 

27.6 

28.0 

17.8 

13.9 

Standard  Deviation 

2.0 

1.7 

1.5 

1.7 

Number  of  Responses 

76 

76 

76 

76 

PREDOMINANTLY  INPUT/OUTPUT 

Average 

20.5 

25.6 

42.3 

10.5 

Upper  Bound 

22.9 

28.1 

45.6 

12.2 

Lower  Bound 

18.0 

23.2 

39.0 

8.7 

Standard  Deviation 

1.5 

1.5 

2.0 

1.1 

Number  of  Responses 

75 

75 

75 

75 

PREDOMINANTLY  INTERACTIVE 

Average 

27.0 

34.2 

26.4 

12.1 

Upper  Bound 

30.1 

37.1 

29.5 

14.1 

Lower  Bound 

24.0 

31.3 

23.3 

10.2 

Standard  Deviation 

1.8 

1.8 

1.9 

1.2 

Number  of  Responses 

75 

75 

75 

75 

PREDOMINANTLY  COMPUTATIONAL 

Average 

27.7 

16.9 

13.3 

41.1 

Upper  Bound 

30.8 

19.2 

15.0 

44.9 

Lower  Bound 

24.6 

14.7 

11.7 

37.2 

Standard  Deviation 

1.9 

1.4 

1.0 

2.3 

Number  of  Responses 

75 

75 

75 

75 

F  -  10 


INHERENT  FACTORS 


LOGIC 


MANY  DISTIICT  OPERATIONAL  MISSIONS 


Average  38 . 5 

Upper  Bound  41 . 7 

Lower  Bound  35.3 

Standard  Deviation  1.9 

Number  of  Responses  74 


SEVERAL  VARIATIONS  OF  OPERATIONAL  MISSIONS 

Average  36.2 
Upper  Bound  39.3 

Lower  Bound  33.0 

Standard  Deviation  1 . 9 

Number  of  Responses  74 


SINGLE  OPERATIONAL  MISSION 

Average  32.3 
Upper  Bound  35.5 

Lower  Bound  29 . 0 

Standard  Deviation  2.0 

Number  of  Responses  74 


MANY  OPERATIONS  REQUIRED  -  HIOILY  INTERRELATED 

Average  38.5 
Upper  Bound  41.7 

Lower  Bound  35.3 

Standard  Deviation  2.0 

Number  of  Responses  78 


MANY  OPERATIONS  REQUIRED  -  RELATIVELY  INDEPENDENT 

Average  37.2 
Upper  Bound  40.6 

Lower  Bound  33.9 

Standard  Deviation  2.0 

Number  of  Responses  75 


INHERENT  FACTORS 


LOGI 


STdia 


FEW  OPE3VVTIONS  REQUIRED  -  HIGHLY  INTERRELATED 

Average 
Upper  Bound 
Lower  Bound 
Standard  Deviation 
Number  of  Responses 


FEW  OPERATIONS  REQUIRED  -  RELATIVELY  INDEPENDENT 

Average  34. 
Upper  Bound  37. 

Lower  Boimd  31 . 

Standard  Deviation  1. 

Number  of  Responses  7 


EXTENSIVE  HARDWARE  INTERFACE  REQUIREMEJJTS 

Average 
Upper  Bound 
Lower  Bound 
Stemdard  Deviation 
Number  of  Responses 


MINIMAL  HARDWARE  INTERFACE  REQUIREMENTS 

Average 
Upper  Bound 
Lower  Bound 
Standard  Deviation 
Number  of  Responses 


EXTENSIVE  SOFTWARE  INTERFACE  REQUIREMENTS 

Average 
Upper  Bound 
Lower  Bound 
Stamdard  Deviation 
Number  of  Responses 


INHERENT  FACTORS 


LOGIC 


INT 


I/O 


COMP 


MINIMAL  SOFTVIARE  INTERFACE  REQUIREMENTS 


Average 

30.7 

27.5 

21.5 

18.9 

Upper  Bound 

33.3 

30.9 

23.3 

21.4 

Lower  Bound 

28.2 

24.2 

19.7 

16.3 

Standard  Deviation 

1.6 

2.0 

1.1 

1.5 

Number  of  Responses 

72 

72 

72 

72 

EXTENSIVE  HUMAN  INTERFACE  REQUIREMENTS 


Average 

27.0 

34.1 

26.7 

11.2 

Upper  Bound 

30.0 

37.6 

30.0 

12.9 

Lower  Bound 

24.0 

30.5 

23.4 

9.4 

Standard  Deviation 

1.8 

2.2 

2.0 

1.1 

Number  of  Responses 

75 

75 

75 

75 

MINIMAL  HUMAN  INTERFACE  REQUIREMEIJTS 


Average 

32.2 

26.9 

21.3 

17.8 

Upper  Bound 

34.9 

29.5 

23.3 

20.1 

Lower  Bound 

29.4 

24.4 

19.2 

15.6 

Standard  Deviation 

1.7 

1.6 

1.2 

1.4 

Number  of  Responses 

72 

72 

72 

72 

WIDE  RANGE  OF  ERROR-PRONE  INPUTS 


Average 

30.3 

23.6 

28.3 

17.8 

Upper  Bound 

34.4 

26.8 

32.2 

21.0 

Lower  Bound 

26.3 

20.3 

24.4 

14.6 

Standard  Deviation 

2.5 

2.0 

2.4 

2.0 

Number  of  Responses 

73 

73 

73 

73 

_SYS42 : :OPAO : .OPERATOR  15:07:36 .73 
SYSTEM  COMING  DOWN  AT  6:00PM  FOR  BACKUPS 
WIDE  RANGE  OF  ERROR-FREE  INPUTS 


Average 

30.4 

25.3 

24.7 

19.2 

Upf)er  Bound 

33.4 

28.0 

27.2 

21.8 

Lower  Bound 

27.4 

22.5 

22.3 

16.5 

Steuidard  Deviation 

1.8 

1.7 

1.5 

1.6 

Number  of  Responses 

70 

70 

70 

70 

INHERENT  FACTORS 


NARROW  RANGE  OF  ERROR-PRONE  INPUTS 


Average 
Upper  Boiind 
Lower  Bound 
Standard  Deviation 
Number  of  Responses 


NARROW  RANGE  OF  ERROR-FREE  INPUTS 


Average 
Upper  Bound 
Lower  Bound 
Standard  Deviation 
Number  of  Responses 


VV. 


AVOIDANCE  MECHANISMS 


LOGIC 


INT 


I/O 


COMP 


INDEPENDENT  C3UALITY  ASSURANCE  ORGANIZATION 

Average 

35.4 

34.7 

30.8 

30.3 

Upper  Bound 

41.2 

40.0 

36.3 

36.1 

Lower  Bound 

29.6 

29.4 

25.3 

24.5 

Standard  Deviation 

3.5 

3.2 

3.3 

3.6 

Number  of  Responses 

70 

69 

66 

67 

INDEPENDENT  TEST  ORGANIZATION 

Average 

30.4 

33.8 

34.9 

35.4 

Upper  Bound 

35.6 

39.1 

40.4 

40.6 

Lower  Bound 

25.1 

28.5 

29.5 

30.1 

Standard  Deviation 

3.2 

3.2 

3.3 

3.2 

Number  of  Responses 

69 

68 

68 

70 

INDEPENDENT  VERIFICATION  AND  VALIDATION  (IV&V) 

Average 

36.9 

33.9 

35.2 

37.8 

Upper  Bound 

42.2 

39.0 

40.7 

43.4 

Lower  Bound 

31.5 

28.8 

29.6 

32.2 

Standard  Deviation 

3.2 

3.1 

3.3 

3.4 

Number  of  Responses 

69 

68 

66 

67 

USE  OF  A  SOFTWARE  SUPPORT  LIBRARY 

Average 

22.1 

24.3 

22.8 

24.3 

Upper  Bound 

26.3 

28.7 

27.0 

28.8 

Lower  Bound 

17.9 

20.0 

18.6 

19.7 

Standard  Deviation 

2.5 

2.6 

2.6 

2.8 

Number  of  Responses 

67 

68 

66 

69 

USE  OF  A  SOFTWARE  CONFIGURATION  CONTROL  BOARD 

Average 

22.5 

32.1 

22.3 

16.9 

Upper  Bound 

27.0 

37.1 

26.6 

20.8 

Lower  Bound 

18.1 

27.1 

18.0 

13.0 

Standard  Deviation 

2.7 

3.1 

2.6 

2.4 

Number  of  Responses 

67 

71 

67 

66 

F  -  15 


AVOIDANCE  MECHANISMS 


LOGIC 


INT 


I/O 


C(»IP 


THOROUGH  AND  ENFORCED  SOFTWARE  DEVELOPMENT  PLAN 


Average 

32.8 

34.8 

33.2 

32.2 

Upper  Bomd 

38.0 

39.8 

38.3 

37.5 

Lower  Bound 

27.7 

29.8 

28.2 

26.9 

Standard  Deviation 

3.1 

3.0 

3.1 

3.2 

Number  of  Responses 

72 

72 

71 

71 

RIGIDLY  CONTROLLED  SYSTEM  REQUIREMENTS  SPEC 

Average 

35.4 

38.3 

32.3 

30.2 

Upper  Bound 

40.3 

43.1 

36.9 

35.2 

Lower  Bound 

30.5 

33.4 

27.6 

25.3 

Standard  Deviation 

3.0 

3.0 

2.8 

3.0 

Number  of  Responses 

75 

73 

72 

69 

RIGIDLY  CONTROLLED  INTERFACE  DESIGN  SPEC 

Average 

30.2 

53.4 

37.0 

23.1 

Upper  Bound 

35.1 

58.3 

42.1 

27.8 

Lower  Bound 

25.3 

48.6 

32.0 

18.3 

Standard  Deviation 

3.0 

2.9 

3.1 

2.9 

Number  of  Responses 

70 

77 

72 

68 

RIGIDLY  CONTROLLED  SOFTWARE  REQUIREMENTS  SPEC 

Average 

40.2 

39.5 

36.6 

34.0 

l^per  Bound 

45.1 

44.3 

41.4 

38.6 

Lower  Bound 

35.3 

34.7 

31.8 

29.5 

Standard  Deviation 

3.0 

2.9 

2.9 

2.8 

Number  of  Responses 

75 

75 

71 

72 

RIGIDLY  CONTROLLED  SOFTWARE  FUNCTIONAL  DESIGN 

SPEC 

Average 

36.3 

38.8 

34.5 

34.4 

Upper  Bound 

40.9 

43.6 

39.3 

39.2 

Lower  Bound 

31.7 

34.0 

29.7 

29.7 

Standard  Deviation 

2.8 

2.9 

2.9 

2.9 

Number  of  Responses 

74 

74 

72 

71 

r 


AVOIDANCE  MECHANISMS 


RIGIDLY  CONTROLLED  SOFTWARE  DETAILED  DESIGN  SPEC 

Average  38 

Upper  Bomd  43 

Lower  Boiind  33 

Standard  Deviation  2 

Number  of  Responses 


REQUIREMENTS  TRACEABILITY  MATRIX 

Average 
Upper  Boiond 
Lower  Boiind 
Standard  Deviation 
Number  of  Responses 


STRUCTURED  ANALYSIS  TOOLS 

_SYS42 ; tOPAO : , OPERATOR  15 : 08:16.11 

PLEASE  DONOT  SUBMIT  ANY  VERY  LONG  JOBS 

Average 
Upper  Bound 
Lower  Bound 
Standard  Deviation 
Number  of  Responses 


PROGRAM  SPECIFICATIOJ  LANGUAGE  (PSL) 

Average 
Upper  Bound 
Lower  Bound 
Standard  Deviation 
Number  of  Responses 


PROGRAM  DESIGN  LANGUAGE  (PDL) 

Average 
Upper  Bound 
Lower  Bound 
Standard  Deviation 
Number  of  Responses 


F  -  17 


w 


AVOIDANCE  MECHANISMS 


V 
■J 

■i 

K’ 

V 

« 


I 


HIGH  ORDER  LANGUAGE  (HOL) 


Average 
Upper  Bound 
Lower  Boiind 
Standard  Deviation 
Number  of  Responses 


HIERARCHICAL,  TOP-DOWN  DESIGN 


Average 
Upper  Bound 
Lower  Bound 
Stamdard  Deviation 
Number  of  Responses 


STRUCTURED  DESIGN 


Average 
Upper  Bound 
Lower  Bound 
Standard  Deviation 
Number  of  Responses 


V 

V 


SINGLE  FUNCTION  MODULARIZATIOJ 


Average 
Upper  Bound 
Lower  Bound 
Standard  Deviation 
Number  of  Responses 


STRUCTURED  CODE 


Average 
Upper  Bound 
Lower  Bound 
Steindard  Deviation 
Number  of  Responses 


AVOIDANCE  MECHANISMS 

LOGI 

C  INT 

I/O 

COMP 

USE  OF  AUTOMATIC  MEASUREMENT  TOOLS 

Average 

23.9 

22.0 

22.8 

24.5 

Upper  Bound 

28.3 

26.4 

27.2 

29.4 

Lower  Bound 

19.4 

17.6 

18.4 

19.5 

Standard  Deviation 

2.7 

2.7 

2.7 

3.0 

Number  of  Responses 

57 

56 

56 

56 

n 


USE  OF  AUTOMATIC  TEST  TOOLS 


1 

Average 

30.1 

28.2 

28.8 

30.8 

Upper  Bound 

35.0 

32.9 

33.5 

35.8 

N 

Lower  Bound 

25.1 

23.4 

24.0 

25.7 

Standard  Deviation 

3.0 

2.9 

2.9 

3.1 

k'-. 

Number  of  Responses 

64 

63 

64 

64 

F  -  19 


s 


m 


DETECTION  MECHANISMS 

LOGIC 

INT 

I/O 

COMP 

FREQUENT  PEER  WALKTHROUGHS 

Average 

45.2 

40.5 

37.9 

39.3 

Upper  Bound 

49.9 

45.4 

42.9 

44.3 

Lower  Bound 

40.5 

35.5 

32.8 

34.3 

Standard  Deviation 

2.9 

3.0 

3.1 

3.0 

Number  of  Responses 

80 

75 

75 

77 

INFREQUENT  PEER  WALKTHROUGHS 

Average 

25.1 

22.5 

19.5 

20.3 

Upper  Bound 

28.7 

26.4 

23.0 

23.7 

Lower  Bound 

21.5 

18.7 

16.0 

16.8 

Standard  Deviation 

2.2 

2.3 

2.1 

2.1 

Number  of  Responses 

72 

70 

69 

70 

FREQUENT  PROGRESS  REVIEWS 

Average 

24.5 

22.0 

21.5 

19.8 

Upper  Bound 

29.1 

26.4 

25.9 

23.9 

Lower  Bound 

19.8 

17.7 

17.0 

15.6 

Standard  Deviation 

2.8 

2.6 

2.7 

2.5 

Number  of  Responses 

68 

68 

67 

68 

INFREQUENT  PROGRESS  REVIEWS 

Average 

11.0 

11.0 

10.2 

8.8 

Upper  Bound 

13.3 

13.5 

12.5 

10.8 

Lower  Bound 

8.8 

8.6 

7.8 

6.7 

Standard  Deviation 

1.4 

1.5 

1.5 

1.3 

Number  of  Responses 

62 

63 

63 

61 

FREQUENT  QUALITY  AUDITS 

Average 

28.8 

26.9 

26.7 

24.5 

Upper  Bound 

33.4 

31.5 

31.3 

29.0 

Lower  Bound 

24.3 

22.2 

22.1 

20.1 

Standard  Deviation 

2.8 

2.8 

2.8 

2.7 

Number  of  Responses 

71 

69 

68 

70 

DETECTION  MECHANISMS  LOG 


INFREQUENT  QUALITY  AUDITS 

Average  13. 
Upper  Bo\ind  16. 

Lower  Bo\and  10 . 

Standard  Deviation  1 . 

Number  of  Responses  6 


USE  OF  SOFTWARE  PROBLEM  REPORTS  PRIOR  PQT 

Average  33 

Upper  Bound  38 

Lower  Bound  28 

Standard  Deviation  3 

Number  of  Responses 


USE  OF  SOFTWARE  PROBLEM  REPORTS  SUBSEQUENT  TO  PQT 

Average  26 

Upper  Bound  30 

Lower  Bound  22 

Standard  Deviation  2 

Number  of  Responses  ( 


USE  OF  SOFTWARE  PROBLEM  REPORTS  SUBSEQUENT  TO  FQT 

Average  24 

Upper  Bound  29 

Lower  Bound  19 

Standard  Deviation  2 

Number  of  Responses  i 


USE  OF  SPECIFICATION  CHANGE  NOTICES  (SCN's) 

Average  19 

Upper  Bound  23 

Lower  Bound  15 

Standard  Deviation  2 

Number  of  Responses 
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DETECTION  MECHANISMS 


LOGI 


USE  OF  ENGINEERING  CHANGE  NOTICES  (ECN's) 


Average 

20.1 

23.1 

19.8 

18.2 

Upper  Bound 

24.5 

27.7 

24.2 

22.5 

Lower  Bound 

15.7 

18.4 

15.4 

13.9 

Standard  Deviation 

2.7 

2.8 

2.7 

2.6 

Number  of  Responses 

61  . 

64 

62 

61 

SOFTVIARE  REQUIREMENTS  REVIEW  (SRR) 


Average 

31.4 

33.1 

28.0 

23.4 

Upper  Bound 

36.3 

37.6 

32.7 

27.9 

Lower  Bound 

26.5 

28.6 

23.4 

18.8 

Standard  Deviation 

3.0 

2.7 

2.8 

2.8 

Number  of  Responses 

73 

72 

69 

67 

PRELIMINARY  DESIC2^  REVIEW  (PDR) 


Average 

28.6 

32.1 

27.9 

22.7 

Upper  Bound 

32.8 

36.1 

32.0 

26.8 

Lower  Bound 

24.5 

28.0 

23.8 

18.5 

Standard  Deviation 

2.5 

2.5 

2.5 

2.5 

Number  of  Responses 

73 

73 

70 

67 

CRITICAL  DESIGN  REVIEW  (CDR) 


Average 

32.3 

32.8 

30.5 

26.2 

Upper  Bound 

36.6 

36.8 

34.8 

30.4 

Lower  Bound 

28.1 

28.7 

26.2 

22.0 

Standard  Deviation 

2.6 

2.5 

2.6 

2.6 

Number  of  Responses 

75 

74 

71 

70 

TEST  READINESS  REVIEW  (TRR) 


Average 

20.1 

23.4 

21.9 

20.3 

Upper  Bound 

24.0 

27.5 

25.9 

24.4 

Lower  Bound 

16.3 

19.3 

17.9 

16.2 

Standard  Deviation 

2.4 

2.5 

2.4 

2.5 

Number  of  Responses 

68 

71 

68 

68 
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DETECTION  MECHANISMS 


LOGIC 


INT 


I/O 


FUNCTIONAL  CONFIGURATION  AUDIT  ( FCA) 


Average 

18.2 

21.8 

20.8 

Upper  Botind 

22.4 

26.1 

25.3 

Lower  Bound 

14.0 

17.5 

16.4 

Standard  Deviation 

2.6 

2.6 

2.7 

Number  of  Responses 

68 

71 

68 

PHYSICAL  CONFIGURATIOI  AUDIT  (PCA) 

Average 

14.9 

17.4 

17.7 

Upper  Bound 

18.7 

21.0 

21.4 

Lower  Bound 

11.2 

13.8 

14.1 

Standard  Deviation 

2.3 

2.2 

2.2 

Number  of  Responses 

67 

69 

66 

INFORMAL  UNIT-LEVEL  TESTING 

Average 

43.4 

31.8 

34.5 

Upper  Bound 

48.1 

37.1 

39.7 

Lower  Bound 

38.6 

26.6 

29.3 

Standard  Deviation 

2.9 

3.2 

3.2 

Number  of  Responses 

77 

73 

74 

PRELIMINARY  QUALIFICATION  TESTING  (PQT) 

Average 

36.1 

34.1 

31.8 

Upper  Bound 

40.7 

38.5 

36.4 

Lower  Bound 

31.4 

29.7 

27.3 

Steuidard  Deviation 

2.8 

2.7 

2.8 

Number  of  Responses 

70 

74 

73 

FORMAL  QUALIFICATION  TESTING  (FQT) 

Average 

31.4 

30.0 

28.9 

Upper  Bound 

36.0 

34.4 

33.7 

Lower  Bound 

26.7 

25.6 

24.2 

Standard  Deviation 

2.8 

2.7 

2.9 

Number  of  Responses 

74 

76 

75 

Average 

34.9 

45.0 

38.2 

33.4 

It  Bound 

39.5 

49.5 

42.8 

38.3 

VVifl 

r  Boiind 

30.4 

40.5 

33.7 

28.4 

viation 

2.8 

2.7 

2.8 

3.0 

m 

m 

sponses 

75 

77 

77 

73 

Average 

33.0 

41.3 

36.7 

31.2 

P-,  „ 

r  Bound 

37.7 

45.9 

41.5 

36.4 

i 

r  Bound 

28.2 

36.8 

32.0 

26.0 

viation 

2.9 

2.8 

2.9 

3.2 

sponses 

77 

79 

79 

74 

m 

Average 

32.7 

35.2 

36.3 

31.2 

r  Bound 

36.0 

38.4 

39.5 

34.6 

r  Bound 

29.4 

32.0 

33.1 

27.8 

viation 

3.3 

3.2 

3.2 

3.4 

sponses 

75 

78 

77 

75 
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^  Rome  Air  Devebpment  Center  ^ 

1  RAVC  plani  and  execu^eA  fid&zaaak ,  dzxj^topmznt,  tz&t  ^ 

and  6zi.zct.zd  accuit^-it-Lon  pAogaam6  tn  6  uppoAt  ^ 

^  Command,  ContAol,  Communlcat-Lom  and  Intzlltgzncz  ? 

%  (C^I)  actto -ittz6 ,  Tzchntcat  and  zngtnzzA-ing  Q 

S  6uppoAt  witkin  aAza6  oi  compztzncz  l6  pAovidzd  to  ^ 

J  ESV  PAogAam  0^^tcz6  (P06}  and  othzA  ESV  zlzmznU  \ 

.  to  pzA^oAm  zilzctloz  acqaliltlon  oi  C^I  6y6tzm6,  A 

S  Tkz  aAza6  OjJ  tzchntcal  compztzncz  inctudz  v 

jfc  communtcatton6 ,  command  and  contAol,  battlz  Ct 

>  management,  In^oAmatton  pAocz66tng,  6uA\Jzlltancz 
^  6zn60A6,  Intzlllgzncz  data  collection  and  handling, 

^  6  olid  itatz  6clzncz6 ,  zlzctAomagnztlc& ,  and 

;  pAopagatlon,  and  zlzctAonlc,  maintainability , 

*  and  compatibility. 


